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Executive Summary 


This report presents the Applied Meteorology Unit's (AMU) evaluation of the current version of the 
Regional Atmospheric Modeling System (RAMS) within the Eastern Range Dispersion Assessment System 
(ERDAS) for the 1999 Florida warm-season months of May-August. Since a systematic evaluation of the 
current version of ERDAS RAMS (hereafter RAMS) had not been performed, representatives from both the 
45th Range Safety (45 SW/SE) and 45th Weather Squadron (45 WS) and expressed a strong interest in 
understanding the accuracy of the current system. RAMS output serves as input to the dispersion model used by 
45 SW/SE. In addition, the 45 WS is interested in using the high-resolution RAMS output as an additional 
forecast guidance tool during routine operations. The primary goal of this task is to determine the accuracy of 
RAMS forecasts during all seasons and under various weather regimes, concentrating on wind and temperature 
forecasts required for dispersion predictions. 

The RAMS is run twice daily with initialization times of 0000 and 1200 UTC. The domain consists of four 
nested grids with horizontal grid spacing of 60, 15, 5, and 1.25 km. The innermost nested grid (grid 4) is 
centered on KSC/CCAFS. RAMS generates initial conditions by using the 12-h forecast grid from the National 
Centers for Environmental Prediction (NCEP) Eta model as a first-guess field, and by analyzing high-resolution 
local data sets including KSC/CCAFS wind tower and profiler data. The model integrates forward in time for 
24 hours using the Eta 12-36-hour forecasts as boundary conditions. Forecast output is available once per hour. 

The evaluation of RAMS for the 1999 Florida warm season is comprised of objective and subjective 
components. The objective component focuses on point error statistics at all observational locations on grid 4 
and selected observations on grids 1-3. The objective component is subdivided into three segments. 

• An evaluation of the full 4-grid configuration of RAMS. This segment calculates 
errors for the current operational configuration of RAMS. 

• A comparison between the full 4-grid configuration and a 3 -grid configuration in 
which RAMS is run with grids 1-3 only. This segment determines the effects of a 
decrease in horizontal resolution on model accuracy. 

• A comparison between the 4-grid configuration of RAMS and the NCEP Eta model. 

This segment determines the benefit that the local high-resolution model can provide 
over the coarser Eta model. 

The subjective component focuses on the verification of the central Florida East Coast Sea Breeze (ECSB) 
and precipitation forecasts on grid 4. The subjective evaluation consists of two segments. The first segment 
involves a verification of the forecast ECSB occurrence and timing to the nearest hour at selected KSC/CCAFS 
wind towers. The second segment verifies the occurrence of forecast precipitation across the domain of grid 4. 
No provisions are used to verify the intensity of precipitation in this report. 

The objective evaluation of the RAMS full, 4-grid configuration initialized at 0000 UTC reveals the 
following significant results. 

• The model generates a distinct cool, dry bias at the surface during the daylight hours. 

• The model root mean square (RMS) errors in surface wind direction are 25-50° and 
consist mainly of random variability and virtually no bias. The largest errors in 
surface wind direction occur during the nocturnal and early morning hours when light 
and variable winds predominate. 

• The maximum model RMS errors in surface wind speed are 2 m s’ 1 during the 
daylight hours with a +1 ms' 1 bias. 

• At the upper levels of the atmosphere, RAMS generates a temperature profile that is 
too stable relative to observations. A cool bias of -1 to -2.5°C occurs at low levels in 
conjunction with a 0.5°C warm bias aloft. The maximum temperature errors are 
found at low levels and smaller errors occur at mid and upper levels. 

• The wind direction RMS errors decrease with height in the atmosphere whereas wind 
speed RMS errors increase with height. The largest wind direction errors occur near 



the surface during the early morning hours and at about 1-2 km above ground level 
during the late afternoon hours. 

• The forecast wind speed exhibits the largest error in the 10-14 km layer where a 
positive bias up to 2 m s _1 occurs before the 6-h forecast, and a negative bias up to -3 
m s' 1 after the 6 h forecast. 

Similar errors are evident in the 1200 UTC forecast cycle with the exception that the low-level temperature and 
dew point temperature errors are slightly smaller. A more sophisticated initialization scheme, such as four- 
dimensional data assimilation, may improve model performance by better incorporating the high-resolution 
observational data of east-central Florida into the RAMS initial condition. 

The objective comparison between the RAMS 4-grid and 3-grid configurations yields mixed results. The 
most notable aspects of the comparison are as follows. 

• The 3 -grid forecasts exhibit a more pronounced daytime cool, dry bias at low levels. 

The typical low-level temperature and dew point temperature errors are nearly 
double in the 3-grid configuration compared to the 4-grid results. 

• Only minor differences occur between the 4-grid and 3-grid forecast wind direction 
and speed. The most substantial variation is that the 3 -grid forecast wind speed is 
virtually unbiased at the surface and experiences a smaller negative bias at upper 
levels than the 4-grid forecasts. 

The objective comparison between RAMS and the Eta model at the Shuttle Landing facility (TTS) and 
Florida rawinsonde sites indicates that errors are comparable in both models with a few exceptions. 

• The Eta model outperforms RAMS in surface temperature forecasts due to the 
prevailing cool bias in RAMS. 

• The Eta model exhibits a +1°C surface dew point temperature bias compared to 
virtually no bias in RAMS. 

• The Eta model has a +1-2 m s“ l wind speed bias compared to little bias in RAMS. 

• At upper levels, both models generate forecast temperature profiles that are too 
stable, but due to the low-level cool bias, the RAMS profile has a greater deviation 
from the observed profile. 

• The upper level wind direction RMS error of the Eta model is about 10-15° smaller 
than RAMS in the 2-7 km layer. 

• The Eta model low-level wind speed across the Florida peninsula is virtually 
unbiased compared to a +1.5 m s’ 1 bias in RAMS. 

The results of the subjective evaluation suggest that RAMS demonstrates a high level of skill in predicting 
the occurrence of the ECSB at the selected KSC/CCAFS wind towers. A small False Alarm Rate (FAR, 0.03) 
combined with a high Probability of Detection (POD, 0.95) support this claim. The timing errors at the selected 
wind towers suggest that RAMS can adequately predict the onset time of the ECSB to the nearest hour, 
comparable to the availability of RAMS forecast output at once per hour. 

The results of the RAMS subjective precipitation verification indicate that the FAR generally exceeds the 
POD, especially for the coastal areas of east-central Florida. The skill scores are more favorable over mainland 
Florida primarily due to a greater frequency of observed precipitation. The 1200 UTC forecast cycle 
experiences slightly higher skill in precipitation forecasts compared to the 0000 UTC forecast cycle. It is 
important to note that anomalous forecast precipitation in RAMS may be the cause of a significant portion of the 
wind and temperature errors, especially during the daylight hours. 

The AMU will extend and enhance the RAMS evaluation for the upcoming 2000 warm season. In 
particular, the AMU will augment the subjective sea breeze verification by including all KSC/CCAFS wind 
towers and developing a more objective technique based on the current methodology. The first thunderstorm of 
the day will be verified on grid 4 in addition to accumulated precipitation. Finally, RAMS forecasts will be 
categorized according to specific meteorological and/or error regimes in order to diagnose model performance 
based on these characterizations. 
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Introduction 


The Eastern Range Dispersion Assessment System (ERDAS; originally called the Emergency Response 
Dose Assessment System) was developed by Mission Research Corporation/ASTER Division (formerly ASTeR, 
Inc.) for the United States Air Force (USAF). ERDAS is designed to provide emergency response guidance for 
operations at the Kennedy Space Center (KSC) and Cape Canaveral Air Force Station (CCAFS) in the event of a 
hazardous material release or an aborted vehicle launch. ERDAS was delivered to the Eastern Range at CCAFS 
in March 1994. Under Applied Meteorology Unit (AMU) option-hours funding from the USAF Space and 
Missile Systems Center (SMC), ENSCO was tasked to evaluate the prototype ERDAS during the period March 
1994 to December 1995. The evaluation report concluded that ERDAS provided significant improvement over 
current toxic dispersion modeling capabilities but contained a number of deficiencies. These deficiencies were 
corrected in the next generation of ERDAS that is part of the newly upgraded Meteorological and Range Safety 
Support (MARSS) replacement system. 

LI Task Background 

The MARSS replacement system contains an upgraded version of the Regional Atmospheric Modeling 
System (RAMS) that is designed to run on workstations with multiple processors. Developed at Colorado State 
University, RAMS is a dynamical numerical weather prediction model with optional parameterization schemes 
for representing physical processes in the atmosphere. The model may be run in two or three dimensions and in 
hydrostatic or non-hydrostatic modes. RAMS features include a terrain-following vertical coordinate, a variety 
of lateral and upper boundary conditions, and capabilities for mixed-phase microphysics. Details on the history, 
overview, and applications of RAMS can be found in Pielke et al. (1992) whereas a description of ERDAS can 
be found in Lyons and Tremback ( 1 994). 

There are two main differences between the original and upgraded versions of the RAMS configuration in 
ERDAS. First, the original configuration of RAMS ran without cloud microphysics whereas the new 
configuration is run with full cloud microphysics on all grids. Second, the areal extent of the innermost, nested 
grid was expanded and the horizontal resolution was improved from 3 to 1.25 km. While the previous 
configuration of ERDAS was validated (Evans et al. 1996), a systematic evaluation of the new configuration of 
ERDAS has not yet been performed. For this reason, representatives from 45th Range Safety (45 SW/SE) and 
45th Weather Squadron (45 WS) requested that the upgraded ERDAS-REPL version of RAMS be evaluated. 

The prognostic gridded data from RAMS is available to ERDAS for display and input to the Hybrid Particle 
and Concentration Transport (HYP ACT) model. The HYP ACT model provides three-dimensional dispersion 
predictions using RAMS forecast grids as initial conditions. Therefore, the accuracy of dispersion predictions 
using the HYP ACT model is highly dependent upon the accuracy of RAMS forecasts. 

The primary goal of this evaluation is to determine the accuracy of RAMS forecasts during all seasons and 
under various weather regimes. The evaluation protocol is based on the operational needs of 45 SW/SE and 45 
WS and is designed to provide specific information about the capabilities, limitations, and daily use of ERDAS 
RAMS for operations at KSC/CCAFS. The ERDAS RAMS evaluation primarily concentrates on wind and 
temperature (stability) forecasts that are required for dispersion predictions using the HYP ACT model. When 
the evaluation is completed in the beginning of 2001, operational users will have tools to verify RAMS in real 
time and information in the form of evaluation results to help interpret and apply forecast data from ERDAS 
RAMS. This report describes the AMU’s evaluation of the RAMS component of ERDAS for the 1999 warm- 
season months of May-August. 

1.2 RAMS Configuration in ERDAS 

In the ERDAS configuration, the three-dimensional, non-hydrostatic mode of RAMS is run on four nested 
grids with resolutions of 60, 15, 5, and 1.25 km (Fig. 1.1). The lateral boundaiy conditions are nudged (Davies 
1983) by 12-36-h forecasts from the National Centers for Environmental Prediction (NCEP) 32-km Eta model 
that have been interpolated onto an 80-km grid. Output from the Eta model is available every 6 h for boundary 
conditions to RAMS. Two-way interactive boundary conditions are utilized on the inner three grids. The 
physical parameterization schemes used in ERDAS RAMS include a microphysics scheme following Cotton et 
al. (1982), a modified Kuo cumulus convection scheme (Tremback 1990), the Chen and Cotton (1988) radiation 
scheme, a Mellor and Yamada (1982) type turbulence closure, and an 1 1 -layer soil-vegetation model (Tremback 
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and Kessler 1985) with fixed soil moisture in the initial condition. The modified Kuo scheme is run on grids 
1-3 whereas the 1.25-km grid 4 utilizes explicit convection. The mixed-phase microphysics scheme is run on 
all four grids. Table 1 . 1 provides a summary of the grid configurations including the dimensions, horizontal and 
vertical resolutions, and time steps as used in the 4-grid configuration of RAMS in ERDAS. 



Figure 1.1. The real-time RAMS domains for the 60-km mesh grid (grid 1) covering much of the 
southeastern United States and adjacent coastal waters, the 15-km mesh grid (grid 2) covering the Florida 
peninsula and adjacent coastal waters, the 5-km mesh grid (grid 3) covering east-central Florida and adjacent 
coastal waters, and the 1.25-km mesh grid (grid 4) covering the area immediately surrounding KSC/CCAFS. 


Table 1.1. A summary of the grid parameters for all four RAMS grids. The model parameters include 
the number of grid points in the x, y, and z directions (nx, ny, and nz), horizontal resolution (dx), minimum 
and maximum vertical resolutions (dzmin and dzmax), and time steps (dt). 


grid 

nx 

ny 

nz 

dx (km) 

dzmin (m) 


dt (s) 

1 

36 

40 

33 

60 

50 

| 

45 

2 

38 

46 

33 

15 

50 


45 

3 

41 

50 

36 

5 

25 


22.5 

4 

74 

90 

36 

1.25 

25 


7.5 


1.3 RAMS Forecast Cycle 

RAMS is initialized twice-daily at 0000 and 1200 UTC using the Eta 12-h forecast grids and operationally- 
available observational data including the CCAFS rawinsonde (XMR), Aviation Routine Weather Reports 
(METAR), buoys, and KSC/CCAFS wind-tower, and 915-MHz and 50-MHz Doppler Radar Wind Profiler 
(DRWP) data. No variational data assimilation or nudging technique is applied when incorporating 
observational data. Instead, RAMS is initialized from a cold start by integrating the model forward in time from 
a gridded field without any balancing or data assimilation steps. Observational data are analyzed onto hybrid 
coordinates using the RAMS Isentropic Analysis (IS AN) package (Tremback 1990). The IS AN hybrid 
coordinate consists of a combination of constant potential temperature (isentropes) and terrain-following (sigma. 
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a) surfaces on which data are analyzed within the RAMS model domain, similar to the NCEP Rapid Update 
Cycle model (Benjamin et al. 1998). 

The ERDAS RAMS forecast cycle is illustrated in Figure 1.2. The RAMS cycle is run in real-time for a 24- 
fa forecast period on a Hewlett Packard (HP) K460 workstation cluster with 1 1 processors. The model run-time 
performance is optimized by using a message-passing interface (MPI) on the 1 1 processors. In MPI, the run- 
time is significantly reduced compared to a single processor because each processor simultaneously performs 
computations on a portion of the domain (Tremback et al. 1998). 

The operational cycle requires approximately 15 minutes to analyze observational data for the initial 
conditions using IS AN and 10-12 h to complete the 24-h forecast cycle. On many occasions when the model 
produced extensive convection, a 24-h forecast took longer than 12 h to complete due to the calculations 
associated with the microphysics scheme. In these instances, the existing RAMS run is terminated before the 
24-hour simulation is completed, and the new simulation begins. Consequently, RAMS data are occasionally 
missing from the 22-24-h forecasts, primarily due to extensive model convection. In the event of a 1 -cycle 
failure, prognostic data are still available from the previous forecast cycle. 

RAMS forecast output is available once per hour for display and analysis purposes. Thus, all portions of 
this model verification study are limited in time to a frequency of 1 hour, regardless of the frequency of 
available observational data. This frequency of model output presents a limiting factor in the verification since 
warm-season weather phenomena in Florida can develop over time scales much shorter than 1 hour (particularly 
convection). Nonetheless, hourly forecast output at high spatial resolution has the potential to provide valuable 
guidance in forecasting warm-season phenomena in east-central Florida. 


RAMS 
Run Time 


Forecast 

Duration 


Wall-clock times 
0015 / 1215 \ 


10-12 hours 10-12 hours 



0000 UTC cycle 1200 UTC cycle 


Figure 1.2. The ISAN/RAMS analysis and forecast cycle. IS AN runs for approximately 15 minutes after 
the model initialization times (0000 and 1200 UTC) and the RAMS 24-h forecast requires about 10-12 h wall- 
clock time to complete each of the 0000 and 1200 UTC forecast cycles during the warm season. 

1.4 Report Format and Outline 

This task requires analysis of a vast amount of model data and graphs. The AMU used an extensive amount 
of observational data to perform a rigorous validation of RAMS. In order to simplify the presentation of the 
objective results, a bulleted summary is provided at the beginning of each section highlighting the key findings. 
Following each summary, a thorough discussion is provided for those readers interested in the details of the 
results. 

The remaining portion of the report is outlined as follows. Section 2 provides the methodology used in the 
verification of RAMS. Section 3 discusses the usage and availability of observational and model data used in 
this study. Section 4 presents the verification of the 1999 warm-season RAMS forecasts, compares the 4-grid 
configuration against a 3-grid configuration of RAMS, and benchmarks the 4-grid RAMS forecasts against the 
Eta model for the 1200 UTC cycle. Section 5 provides a preview of the additions and improvements that will 
take place in the upcoming 2000 warm-season evaluation. Finally, Section 6 summarizes the key points in the 
validation of RAMS for 1999 warm-season months. 
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2. Methodology 


The AMU evaluation of RAMS during the 1999 warm season includes an objective and subjective 
component. The objective component is designed to present a representative set of model errors of winds, 
temperature, and moisture for both the surface and upper-levels. The goal of the subjective verification is to 
provide an assessment of the forecast timing and propagation of the east-central Florida East Coast Sea Breeze 
(ECSB) and daytime forecast precipitation through an examination of RAMS forecast fields. 

2.1 Objective Component 

The objective component consists of three segments to compute point error statistics. The first segment 
verifies the full 4-grid configuration of RAMS, the second segment compares point statistics between the full 
configuration and a configuration of RAMS with a coarser horizontal resolution, and the third segment compares 
RAMS errors to the Eta model errors. Each portion of the objective component focuses on point error statistics 
at many different observational locations on all four forecast grids. Zero to 24-h point forecasts of wind, 
temperature, and moisture were compared with surface METAR and buoy stations, the XMR rawinsonde, and 
KSC/CCAFS wind-tower, 915-MHz, and 50-MHz DRWP data at all available observational locations on grid 4, 
and selected surface and rawinsonde stations on grids 1-3. The locations of all the observations used for point 
verification are given in Figure 2.1. 

The point statistics presented include the root mean square (RMS) error, bias, and error standard deviation 
(SD) of wind direction, wind speed, temperature, and dew point temperature (dew point) for May- August 1999. 
In addition, the average values of forecasts and observations for these variables were computed as a function of 
forecast hour at all observational sites for the entire four-month evaluation period. Special care was exercised 
when computing the mean and SD of wind direction errors following Turner (1986). However, in general, the 
mean seasonal observed and forecast wind direction quantities have little meaning because the distributions of 
the wind direction were nearly uniform. Therefore, only plots of RMS error and bias are provided for wind 
direction. Error statistics for all other variables were calculated in the manner as outlined below. 

If <t> represents any forecast variable, then forecast error is defined as O’ = O f - <X> 0 , where the subscripts f 
and o denote forecast and observed quantities, respectively. The bias represents the average model error and is 
computed as 

1 N 

Bias = -2>' ; (2.1) 

where N is the total number of forecasts. The RMS error is calculated as 


RMSerror = jlg(<D') 2 . (2.2) 

By applying the Murphy (1988) decomposition for RMS error, the SD of the errors were computed using the 
equation 

SD = ~J RMS 2 - Bias 2 . (2.3) 

A quality control (QC) check was performed on all point error statistics to remove any errors greater than 3 
standard deviations from the mean error (bias). This QC check generally resulted in the rejection of less than 2 
percent of all possible errors. 

The task protocol for the objective component also called for a comparison between analysis and forecast 
gridded data of winds at 16.5 m over the area of grid 4 covered by the KSC/CCAFS wind tower network. An 
automated tool was developed to archive forecast grids of temperatures and winds from the twice-daily RAMS 
simulations. Unfortunately, the grid extraction routine was not fully implemented until mid-July, thereby 
limiting the representativeness of gridded difference fields for the 1999 warm season evaluation. Statistics 
generated from gridded difference fields will be presented only in the RAMS final report. 


4 


It is important to note that this report does not attempt to address the possible causes for the model errors 
presented. This paper only illustrates and interprets the error statistics. However, an ongoing AMU option 
hours task will examine the possible causes for certain type of RAMS systematic errors through sensitivity 
experiments. 






Figure 2.1. A display of the surface and upper-air stations used for point verification of RAMS on all four 
forecast grids. Observational data are used to verify RAMS forecasts according to the finest resolution grid in 
that the stations are located. The observational data used for verification include surface METAR stations (‘X’), 
buoys (filled diamond), rawinsondes (filled square), KSC/CCAFS 915 MHz DRWP (filled triangles in panel d), 
the KSC/CCAFS 50 MHz DRWP (open star in panel d), and Eta point forecast locations (filled star). The 
locations of the inner nested grid within its parent grid is shown in panels a), b), and c). 
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2.2 Benchmark Experiments 

The second and third segments of the objective evaluation consist of two additional benchmark 
experiments. The first benchmark compares the RAMS 4-grid configuration to a 3-grid configuration by simply 
excluding grid 4 and rerunning RAMS only with grids 1, 2, and 3. The statistics were computed separately for 
the 3-grid and 4-grid data for the forecast times in which both models runs were available (refer to Appendix A). 
Section 4.1.2 presents the results of this comparison for observational data within the grid 4 domain. The goal 
of this experiment is to measure the impact from a decrease in horizontal resolution of the innermost grid on the 
resulting forecast errors. 

The second benchmark experiment compares the 32-km Eta point forecasts at 14 surface stations in the 
southeastern United States and 4 upper-air stations in the Florida peninsula to RAMS forecasts interpolated to 
the same stations (refer to Fig. 2. 1 and Table 2. 1). This important benchmark compares RAMS forecasts to the 
widely used NCEP Eta model and quantifies any added value that may be provided by the RAMS model over 
the Eta model. The results from the Eta model benchmark are discussed in Section 4.1.3. 


Table 2.1. A list of the 14 stations in the southeastern United States used to 
compare the RAMS and Eta model error statistics. Each station is listed according to 
the finest-resolution RAMS grid in which it is located. 

Grid 

Stations 

Grid 1 

Athens, GA (AHN) 
Nashville, TN (BNA) 
Greensboro, NC (GSO) 
Hatteras, NC (HSE) 
Tallahassee, FL (TLH) 
Lake Charles, LA (LCH) 

Grid 2 

Cross City, FL (CTY) 
Fort Myers, FL (FMY) 
Jacksonville, FL (JAX) 
Miami, FL (MIA) 

West Palm Beach, FL (PBI) 
Tampa Bay, FL (TPA) 

Grid 3 

Orlando, FL (MCO) 

Grid 4 

Shuttle Landing Facility, FL (TTS) 


2.3 Subjective Component 

The subjective evaluation verifies RAMS forecasts of the onset and movement of the ECSB, precipitation, 
and low-level temperature only on grid 4, The subjective verification was conducted on grid 4 for both the 0000 
UTC and 1200 UTC forecast cycles, but only during working days and for model runs that terminated normally. 

2.3.1 Sea Breeze Verification 

The AMU developed a methodology to verify the timing of the forecast ECSB to the nearest hour at 13 
selected KSC/CCAFS observational towers shown in Figure 2.2. The AMU chose these 13 towers in order to 
verify the propagation of the ECSB from east to west along three separate east-west-oriented lines. The 
subjective methodology used both Geostationary Operational Environmental Satellite (GOES)-8 imagery and 
Weather Surveillance Radar-1988 Doppler (WSR-88D) data to identify the occurrence of the ECSB. To 
determine the timing of the sea breeze passage, the AMU examined each coastal KSC/CCAFS tower for a shift 
to an onshore wind component (wind direction between 330° and 150°) and inland KSC/CCAFS towers for the 
development and maintenance of a wind shift from a westerly to an easterly component. During easterly flow 
regimes, a sea breeze passage was determined by an increase in the negative (easterly) u-wind at each 
KSC/CCAFS tower. The AMU then applied these same wind criteria to the ERDAS RAMS forecasts 
interpolated to each KSC/CCAFS tower location in order to determine the forecast ECSB passage. 
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Figure 2.2, A plot of the 13 KSC/CCAFS wind towers used for the ECSB subjective verification during 
the months of May-August 1999. The stations were chosen to examine the timing of the ECSB along the 
immediate Atlantic coastline, Merritt Island, and mainland Florida. 

A 2 x 2 contingency table is used to summarize the ECSB verification statistics based on the occurrence of 
both an observed and forecast ECSB at any of the 13 KSC/CCAFS towers. A “hit 1 * is defined as the occurrence 
of both an observed and forecast sea breeze passage at any KSC/CCAFS tower. Because RAMS forecast output 
is available once per hour, the AMU verified the timing of the onset and movement of the sea breeze to the 
nearest hour at each of the 13 KSC/CCAFS towers. Table 2.2 is a sample 2x2 contingency table from which a 
variety of categorical and skill scores can be computed to measure the forecast performance. The total number 
of correct forecasts is given by x in the upper left comer (forecast and observed = yes) and w in the lower right 
comer (forecast and observed = no). The number of forecast failures is given in the lower left portion of the 
table (forecast = no, observed = yes) and the number of false alarm forecasts is given in the upper right comer 
(forecast = yes, observed = no). 


Table 2.2. A generic 2x2 contingency table for the evaluation of a 
forecast element is shown from which categorical and skill scores are computed 
(see text). 


Observed = Yes 

Observed = No 

Forecast = Yes 

x 

z 

Forecast = No 

i 

w 

Number of correct forecasts = (x+w) 
Number of false alarm forecasts = z 


1 Number of forecast misses 

= y 



From the contingency table, the AMU calculated categorical and skill scores as defined in Schaefer (1990) 
and Doswell et al. (1990). These scores include the bias, probability of detection (POD), the false alarm rate 
(FAR), critical success index (CSI), and the Heidke skill score (HSS). Using the variables in Table 2.2, these 
scores are defined as follows: 
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x + z 

Bias = . 

(2.4) 

x + y 


POD = * , 

(2.5) 

x + y 


FAR = Z , 

(2.6) 

x + z 


CSI = * , 

(2.7) 

x + y + z 


2 (xw- yz) 

(2.8) 


y 2 + z 2 + 2xw + (y + z\x + w) 


Given the occurrence of a weather element, the POD is the percentage of time that RAMS correctly 
forecasted that element. The FAR is the percentage of time that RAMS incorrectly forecasted a weather element 
when none occurred. The CSI measures the ratio of the number of hits to the number of events plus the number 
of false alarms. The HSS provides a benchmark of the model performance compared to random forecasting 
(HSS=0). Higher POD, CSI, and HSS combined with a low FAR are associated with better performance in the 
model forecasts. In a perfect forecast, the POD, CSI, and HSS are equal to 1 whereas the FAR is 0. Section 
4.2.1 provides a summary of the error statistics and categorical and skill scores for the subjective verification of 
the forecast ECSB. The error statistics generated for the sea breeze timing verification include RMS error, SD, 
and bias in hours in addition to the categorical and skill scores. 

2.3,2 Precipitation Zone Verification 

The subjective verification scheme for precipitation follows the methodology described in Manobianco and 
Nutter (1999). Grid 4 is divided into six separate zones to identify locations of forecast precipitation (Fig. 2.3). 
Hourly forecast precipitation rates are identified from both the 0000 and 1200 UTC RAMS runs in each 
verification zone for the forecast hours valid from 1300 to 0200 UTC (primarily daylight hours). The 
verification technique uses one-hour precipitation rates derived from WSR-88D data to verify the location of 
precipitation. A “hit” is defined as the occurrence of both forecast and observed precipitation within a specific 
zone at a given hour, regardless of the intensity. 

The technique also extends the precipitation verification to two and three-hourly verification bins in order to 
identify any improvement in precipitation forecast skill with a larger verification time interval. Increasing the 
verification time interval and reapplying skill-score thresholds can determine the predictable limit of ERDAS 
RAMS precipitation forecasts. For the two-hour forecast bins, a hit is defined as the occurrence of observed and 
forecast precipitation in the same zone for either hour of the two-hour bin. Similarly for three-hourly bins, a hit 
is defined as the occurrence of both observed and forecast precipitation at any hour within the three-hour time 
interval. To measure the accuracy of RAMS hourly precipitation forecasts, the AMU computed the POD, FAR, 
CSI, and HSS for each zone in Figure 2.3. 

In traditional operational precipitation verification techniques, threat scores are computed for each 
individual grid point to measure the skill of precipitation forecasts at specific intensity thresholds (e.g. 0.01”, 
0.25”, etc.). The precipitation forecast must be accurate in both space and time to receive a high threat score 
according to these traditional techniques (Olson et al. 1995). Traditional threat scores do not usually account for 
spatial errors in forecast precipitation. 

In this study, precipitation is verified according to the occurrence of precipitation (> 0.01”) anywhere within 
the six separate zones of Figure 2.3. Thus, the methodology used in this study is less stringent than threat score 
techniques in terms of spatial and intensity verification. However, this methodology is more stringent 
temporally based on the small time windows used to verify the forecast precipitation (1-3 h). Most current 
operational techniques still verify model precipitation for 24-h periods; however, NCEP recently began routine 
3-h precipitation verification for many national-scale operational models (Baldwin 2000). Section 4.2.2 
summarizes the categorical and skill scores of RAMS forecast precipitation and includes verification scores for 
1-, 2-, and 3-h forecast windows. 
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Figure 2.3. A plot of the 6-zone classification scheme used for the warm-season subjective precipitation 
verification during the warm-season months of June-August 1999. The division between the western (1-3) and 
eastern zones (4-6) is designed to parallel the east-central Florida east coast. 
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3 . 


Data Availability 


The RAMS evaluation during the 1999 warm season was contingent upon the availability and quality of 
observational data and successful model forecasts. This section discusses the general availability of 
observational and model data during the 1999 warm-season study and provides some explanations for loss of 
specific data types at certain times. 

3.1 Observational Data 

As part of the objective verification, the AMU computed statistics for RAMS forecasts at selected surface, 
upper-air, and buoy stations on grids 1-3, and all available observational locations on grid 4 (see Fig. 2.1). 

Among the KSC/CCAFS data sets, the wind tower data were the most reliable, providing more than 3000 
observations of temperatures and winds at most RAMS forecast hours during the four-month period (not 
shown). Observed temperatures were available at both 1.8 m and 16.5 m (54 ft), whereas winds were available 
at 12-ft and 54-ft. 

The XMR rawinsonde provided the only platform for verifying RAMS forecast thermodynamic variables as 
a function of height on grid 4. During the 1999 warm-season months, XMR rawinsondes were routinely 
launched between 1000-1 100, 1500-1600, and 2200-2300 UTC. As a result, the forecast hours valid at these 
times contain the greatest number of observations necessary for a valid upper-air verification of thermodynamic 
and kinematic variables. XMR rawinsonde observations were also taken during all 1999 warm-season launch 
operations at various non-standard times of the day, but generally, there are not enough observations to provide 
a representative verification data set at non-standard times. Therefore, this report presents only the XMR 
verification results valid at 1000, 1500, and 2200 UTC. 

The 50-MHz DRWP offered a good data source for high temporal resolution winds above 2 km; however, 
these data were occasionally of poor quality during active convection. Thus, the actual number of observations 
at each gate of the 50 MHz DRWP is sometimes significantly less than the theoretical maximum due to quality- 
controlled (QC) or missing data. However, between 60-80 % of the theoretical maximum of 50-MHz wind 
observations are available from 2—15 km during forecast hours 0—21. 

The 915-MHz profilers offered the least amount of QC observations that could be used for verification of 
winds below 4 km. RAMS forecasts had to be verified at each 915 MHz profiler site separately because each 
profiler's range gates are at different vertical levels, preventing the profiler verification data to be grouped 
together. In addition to the range gate problem, only one 915 MHz profiler (0001) provided enough quality 
observations at any time for the resulting verification to be considered marginally representative of the 1999 
warm-season months. Even so, the available data at 0001 represents less than half of the maximum number of 
possible observations. Due to the sparse data sets, this report does not present wind verification at any of the 
915 MHz profilers. 

The remaining observational data sets used for verification include the Meteorological Interactive Data 
Display System (MIDDS) national rawinsonde data set, METAR reports, and buoy data. The national 
rawinsonde data set consists of the four national rawinsondes in the Florida peninsula other than XMR (Key 
West, EYW/72201; Miami, MIA/72202; Tampa Bay, TBW/72206; and Jacksonville, JAX/72206). These 
rawinsondes (hereafter referred to as national rawinsondes) provided a representative amount of observations at 
mandatory levels from 1000—100 mb for winds, 1000—400 mb for temperatures, and 1000—500 mb for dew 
points. 

The AMU grouped all METAR stations according to the RAMS grid in which the station is located. For 
example, METAR stations on grid 4 belong to the grid 4 list, stations on grid 3 outside the grid 4 domain belong 
to the grid 3 list, and so forth. The availability of METAR data is generally abundant except for a few specific 
times when very few METAR data were archived. For the 0000 UTC cycle, an insufficient amount of 
observational data were extracted at 0000, 0100, and 1200 UTC (0-h, 1-h, and 12-h valid forecast times 
respectively). For the 1200 UTC cycle, an insufficient number of METAR data were extracted at 0000 and 
0100 UTC (12 and 13-h forecast times). The lack of archived METAR data at these times resulted from a 
conflict between the data archiving programs and the model initialization procedures. 


10 


The two buoys offshore of Cape Canaveral were used (41009 and 41010). No data were available from 
these sites every three hours (0000, 0300, 0600 UTC, etc.). At all other times, between 60-80% of the 
theoretical maximum amount of data were available. 

3.2 Forecast Data 

The 4-grid configuration of RAMS ran routinely during all four warm-season evaluation months. 
Occasionally RAMS forecasts were not produced primarily due to lack of Eta gridded data from MIDDS. 
RAMS requires the Eta 12-h forecast grid as a background field for the model initial condition. In addition, the 
Eta 12-36-h forecast grids supply the boundary conditions for the RAMS 24-h integration. When any of these 
Eta grids were not available from MIDDS, the RAMS model could not be run. The non-availability of Eta grids 
was the most common cause of a RAMS forecast cycle failure. 

Another aspect of the 4-grid configuration limited the amount of verification data primarily for the 22-24-h 
forecast times. Forecast cycles in which RAMS generated a substantial amount of precipitation often did not 
complete within the allotted 12 hours (see Section 1.3). Because of this limitation, the amount of available data 
for both the 0000 UTC and 1200 UTC cycles drops off rapidly after the 22-h forecast. 

Beginning on 1 1 June, the 3-grid configuration of RAMS ran in real-time in conjunction with the 4-grid 
forecasts, using a separate workstation. Three-grid forecasts did not complete on occasions due to both the lack 
of Eta grids and a problem in the hardware that caused the workstation to reboot periodically. 

The third type of forecast data consists of Eta point forecasts that are distributed through MIDDS from the 
Spaceflight Meteorology Group (SMG) at the Johnson Space Center in Houston, TX. The AMU archived 1200 
UTC Eta point forecasts of surface and upper-level data from 14 METAR stations in the southeastern United 
States including the Shuttle Landing Facility (TTS, see Fig. 2.1 and Table 2.1). For verification purposes, 
forecast data from the Eta vertical levels were interpolated to mandatory pressure levels. A complete list of all 
available 0000 and 1200 UTC forecast data for the RAMS 4-grid and 3-grid configurations, and the Eta point 
forecasts is summarized in Appendix A for all four warm-season months. 
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4. 


Results from May-August 1999 


This section presents results from both the objective and subjective components of the RAMS 1999 warm- 
season evaluation. Only the most significant errors and features at select observational sensors are discussed in 
this section. Additional graphs and plots summarizing the objective errors at other selected sensor types are 
found in Appendices B-D. 

4.1 Objective Results 

In this section, average quantities and point error statistics are presented for the 1999 warm-season months 
(May-August). Cumulative results are shown for the entire period rather than for each individual month. For 
purposes of interpretation, total model error (RMS error) includes contributions from both systematic and non- 
systematic errors. Systematic error (bias) can be caused by a consistent misrepresentation of physical 
parameters such as radiation or model-generated convection. Nonsystematic errors given by the error SD 
represent the random errors caused by uncertainties in the model initial condition or unresolveable differences in 
scales between the forecasts and observations. 

4.1.1 Four-Grid Results 


4.1.L1 Summary of Surface (0000 UTC cycle , KSC/CCAFS wind towers, METAR , buoys) 


Table 4.1. Summary of the 4-grid RAMS errors at the KSC/CCAFS wind towers for the 0000 
UTC forecast cycle. Temperature and dew point errors are valid at 1.8 m whereas wind direction and 
speed errors are valid at 16.5 m. 

Variable 

RMS Error 

Bias 

Notable Errors 

Temp 

(°C) 

1 to 4.5 

-3.5 to 0.5 

• Maximum RMS error occurs during the day, composed 
of a -3. 5°C cool bias. 

• The cool bias also occurs over the ocean (at offshore 
buoys). 

• Nocturnal RMS error < 2°C with a slight warm bias. 

• RMS error is 1°C smaller at 16.5 m compared to 1.8 m. 

Dew 

Point 

(°C) 

1 to 3 

-2 to 0.5 

• Maximum RMS error of 3°C occurs during the day, 
composed of a -2°C dry bias. 

• Nocturnal RMS error < 2°C. 

Wind 

Direction 

(deg.) 

15 to 65 

-5 to 5 

• Maximum RMS error of 60-65° occurs during the 
nocturnal and early mornings hours. 

• Daytime RMS error of 40-55°. 

• Observed variability of 16-22° within the tower network 
(Merceret 1995) yields a model error range of 25-50°. 

Wind 

Speed 

( ms '‘) 

1.5 to 2.5 

-0.5 to 1.5 

• RMS error of 1.5 to 2.5 m s’ 1 . RMS error is largest 
during the day when a +1 ms' 1 bias occurs. 




















4.1. 1.2 Summary of Upper Air (0000 UTC cycle, XMR 11-h and 22-h forecasts) 


Table 4.2. Summary of the 4-grid RAMS errors at XMR for the 0000 UTC forecast cycle. 

Variable 

RMS Error 

Bias 

Notable Errors 

Temp 

(°C) 

1 to 3 

-2.5 to 1 

• RAMS consistently generates a forecast temperature 
profile that is too stable compared to XMR. 

• A cool bias occurs below 650 mb whereas a warm bias 
occurs above 650 mb at the 11-h forecast time. 

• RMS error decreases with height from 3°C near the 
surface to about 1°C at 300 mb. 

Dew 

Point 

(°C) 

0.5 to 5 

-1 to 1 

• RMS error is smallest near the surface and upper levels 
and largest at mid levels. 

• Maximum bias (+ 1°C) near the surface and at 650 mb 
in the 11-h forecast. 

• Bias approaches -1°C at low levels during the day. 

Wind 

Direction 

(deg.) 

30 to 70 

-20 to 10 

• At 1 1 h, the largest RMS error occurs near the surface 
due to the light and variable winds. 

• RMS error decreases with height at mid and upper levels 
(especially above 600 mb). 

• At 22 hours, the RMS error peaks at 850 mb above the 
top of the mean observed sea breeze boundary. 

Wind 
Speed 
(m s' 1 ) 

2 to 4 

-1 to 1.5 

• RMS error increases nearly linearly from the surface to 
300 mb. 

• Negative bias occurs at upper levels above 900 mb. 


4.1. 1.3 Summary of Upper Air (0000 UTC cycle, 50-MHz DRWP) 


Table 4.3. Summary of the 4-grid RAMS errors at the KSC/CCAFS 50-MHz DRWP for the 0000 

UTC forecast cycle. 

Variable 

RMS Error 

Bias 

Notable Errors 

Wind 

Direction 

(deg.) 

20 to 60 

-15 to 5 

• Largest RMS error occurs in the 2-6-km range and the 
smallest RMS error is found above 8 km. 

• Forecast wind direction is nearly unbiased for all times. 

Wind 
Speed 
(m s' 1 ) 

2 to 7 

-3 to 2 

• In the first few hours, a positive wind speed bias occurs 
in the 10-14-km layer. 

• After 6 h, the strength of the upper-level winds is under- 
forecasted in the 10-14-km layer. 
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4. 1.1. 4 Detailed Discussion 


a. Surface (0000 UTC cycle at KSC/CCAFS wind towers, METAR stations, buoys) 

1. Temperature and dew point 

The most notable systematic model error in the RAMS 4-grid simulations is a surface and lower- 
tropospheric cool temperature bias that is especially prevalent on the inner three grids. The surface-based cool 
bias manifests itself during the daylight hours when RAMS does not sufficiently warm the temperatures at the 
lowest model vertical level. Figure 4.1 illustrates the cool daytime temperature bias in the 0000 UTC RAMS 
cycle that develops within the KSC/CCAFS wind tower network at the 1.8-m level. This figure displays 
meteograms of the mean forecast and observed temperatures, RMS error, bias, and error SD as a function of 
forecast hour. 

In Figure 4.1a, the mean forecast and observed temperatures are quite similar until the 1 1-h forecast (1100 
UTC), after which the mean forecast temperature is consistently about 2-3 °C cooler than the observations. The 
RMS error gradually increases to 2°C during the overnight hours and then rapidly increases to 4.5°C after the 
12-h forecast (Fig. 4.1b). The discontinuity in RMS error from 1 1-12 h is caused by the sudden increase in 
observed temperatures after sunrise in conjunction with little warming in the forecast temperature. The daytime 
RMS error is composed of a -3.5°C systematic error (compare with bias in Fig. 4.1c) and a 2°C nonsystematic 
error (given by SD in Fig. 4. Id). 

The same type of trends in model errors also occur in the 1200 UTC RAMS forecast cycle (Appendix B, 
Fig. Bl). The primary difference between the 0000 and 1200 UTC forecast cycles is the magnitude decrease in 
the cool bias during the 1200 UTC cycle, peaking at about -2°C at the 10-h forecast. The cool bias gradually 
disappears during the overnight hours ( 1 3-22-h forecast hours) but rapidly develops again between the 23-h and 
24-h forecasts, after sunrise. 

It is imp ortant to note that the cool temperature bias not only occurs over land, but also over the nearby 
Atlantic coastal waters (Fig. 4.2). The temperature errors at the two buoys offshore of the central Florida east 
coast indicate a cool bias on the order of -2°C (Fig. 4.2c). This bias develops during the nocturnal hours, unlike 
the temperature bias over land. However, the cool bias at the buoys peaks during the afternoon hours in a 
similar manner to the land verification (Figs. 4.1c and 4.2c). This result has important implications regarding 
the formation and propagation of the central Florida east coast sea breeze. Because the cool bias is found over 
both the land and nearby ocean, the thermal gradient between the land and ocean that drives the sea breeze 
circulation is not likely affected. 

It is interesting to note that the magnitude of the model cool bias at the KSC/CCAFS wind towers is larger 
at the 1.8-m level rather than at the 16.5-m level. Figure 4.3 shows a meteogram of average forecast and 
observed temperatures and model errors at 16.5 m. For the 0000 UTC cycle, the maximum RMS error at 16.5 m 
(Fig. 4.3b) is about 1.0°C less than maximum RMS error at 1.8 m (Fig. 4.1b). This reduction in RMS error at 
16.5 m primarily results from a decrease in the magnitude of the cool daytime temperature bias (see Figs. 4.3c 
and 4.1c). The smaller temperature RMS error at 16.5 m could be caused by one of two conditions. First, the 
lowest physical vertical level in the RAMS configuration is at 1 1 m (36 ft), which is closer to 16.5m than 1.8 m. 
Thus, the near-surface RAMS temperature forecasts may be more representative of the observed 16.5-m 
temperatures than the 1.8-m temperatures. Second, a combination of the RAMS surface physics, turbulent 
mixing, or radiation schemes may be causing the forecast temperatures to be too cool in the lowest layer of the 
model. 

The results presented in Snook et al. (1998) indicate a similar pattern in the cool daytime temperature bias 
during real-time RAMS simulations over the southeastern United States in support of the 1996 summer Olympic 
games. Their sensitivity experiments suggest that RAMS is slow in mixing out the boundary layer during the 
late morning hours. In contrast, a recent study by Salvador et al. (1999) showed a warm daytime temperature 
bias in RAMS simulations at two coastal locations in Spain. Future investigation is necessary to isolate the 
possible cause(s) for the cool temperature bias in this warm-season study. 

In addition to a surface cool bias, RAMS also exhibits a negative (dry) dew point bias during the daylight 
horns. In the 0000 UTC forecast cycle, the mean observed and forecast dew point at 1.8 m are quite similar 
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until forecast hour 12, after which the mean forecast dew point is lower than the observed values (Fig. 4.4a). 
The maximum RMS error reaches 3°C by the 13-h forecast, and gradually decreases thereafter (Fig. 4.4b). The 
total model error in dew point is composed of a -2°C bias (Fig. 4.4c) and a 2°C error SD (Fig. 4.4d). Similar 
patterns in the dew point errors are evident at the 16.5-m level and in the 1200 UTC cycle, but the maximum 
bias in both cases is less than the 0000 UTC forecast cycle at 1.8 m (not shown). Thus, the temperature and dew 
point forecasts collectively suggest that the RAMS model exhibits surface-based cool and dry biases during the 
daytime hours and smaller errors during the nighttime hours. The AMU will conduct further investigations on 
specific case studies in order to isolate the cause(s) of this surface-based cool, dry bias on grid 4. 

An intriguing aspect of the RAMS statistics is the difference in the temperature and dew point errors for 
surface (METAR) stations on grid 1 versus stations on grid 4 (see Fig. 2.1 for station locations). In order to 
compare the grid errors, the AMU computed separate model errors for all METAR reporting stations located on 
grid 4 and those located only on grid 1 (as well as those stations located on grids 2 and 3). Due to the larger 
domain, more METAR stations were available on grid 1 than grid 4. The temperature errors for grid 4 and grid 
1 METAR stations from the 0000 UTC forecast cycle are given in Figures 4.5 and 4.6, respectively. The 
patterns and magnitudes of the errors for the grid 4 METAR stations are quite similar to the KSC/CCAFS wind 
tower errors. The mean forecast temperature is close to the mean observed temperature until the 13-h forecast, 
after which the mean forecast temperature is as much as 3°C less than the mean observed temperature (Fig. 
4.5a). The maximum RMS error is about 4°C comprised of a -3°C bias and a 2-3°C error SD (Fig. 4.5 b-d). 

In contrast to the grid 4 results, a pronounced nocturnal warm bias develops by forecast hour 3 in the 
RAMS model at the grid 1 METAR stations and decreases during the daylight hours after 12 h. There is little 
evidence for a substantial daytime cool bias on grid 1, contrary to grid 4. The grid-1 nocturnal warm bias is 
evident in the mean observed and forecast temperature traces as well as the bias graph (Figs. 4.6a and c, 
respectively). The RMS error for temperature on grid 1 is generally 2-3 °C, comprised of a warm bias up to 
2°C and an error SD of about 2°C (Fig. 4.6b-d). Based on the 1200 UTC RAMS cycle results on grid 1 
(Appendix B, Fig. B2), a slight cool daytime bias still occurs (-1 °C), similar to the grid 4 results. Thus, the 
reduction of the 0000 UTC cycle warm bias on grid 1 after 12 h may result from the cool daytime bias offsetting 
the warm nocturnal bias. Thus, the daytime cool bias occurs implicitly in the grid-1 bias plot of Figure 4.6c, but 
masked by the prevalent nocturnal warm bias. It is interesting to note that a nocturnal positive (moist) bias in 
dew point occurs on grid 1 in conjunction with the nocturnal warm bias (Appendix B, Fig. B3). 
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0000 UTC 4-grid Cycle Temperature (C) for 1.8-m towers 
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Figure 4.1. A meteogram plot of temperature errors (°C) for the 1999 warm season months for the 4-grid 
configuration of the 0000 UTC RAMS forecast cycle, verified at the 1.8-m level of the KSC/CCAFS wind-tower 
network. Parameters that are plotted as a function of forecast hour include: a) mean observed temperatures 
(dashed) and mean forecast temperatures (solid), b) RMS error, c) bias, and d) error standard deviation (SD). 
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0000 UTC 4-grid Cycle Temperature (C) for FL buoys 
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Figure 4.2. A meteogram plot of temperature errors (°C) for the 1999 warm season months for the 4-grid 
configuration of the 0000 UTC RAMS forecast cycle, verified at the two buoys offshore of KSC/CCAFS. 
Parameters that are plotted as a function of forecast hour include: a) mean observed temperatures (dashed) and 
mean forecast temperatures (solid), b) RMS error, c) bias, and d) error standard deviation (SD). 
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0000 UTC 4-grid Cycle Temperature (C) for 16.5-m towers 
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Figure 4.3. A meteogram plot of temperature errors (°C) for the 1999 warm season months for the 4-grid 
configuration of the 0000 UTC RAMS forecast cycle, verified at the 16.5-m level of the KSC/CCAFS wind- 
tower network. Parameters that are plotted as a function of forecast hour include: a) mean observed 
temperatures (dashed) and mean forecast temperatures (solid), b) RMS error, c) bias, and d) error standard 
deviation (SD). 









0000 UTC 4-grid Cycle Dew point (C) for 1.8-m towers 



Forecast Hour 


Figure 4.4. A meteogram plot of dew point errors (°C) for the 1999 warm season months for the 4-grid 
configuration of the 0000 UTC RAMS forecast cycle, verified at the 1.8-m level of the KSC/CCAFS wind-tower 
network. Parameters that are plotted as a function of forecast hour include: a) mean observed dew points 
(dashed) and mean forecast dew points (solid), b) RMS error, c) bias, and d) error standard deviation (SD). 
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0000 UTC 4-grid Cycle Temperature (C) for grid 4 Sfc. 
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Figure 4.5. A meteogram plot of temperature errors (°C) for the 1999 warm season months for the 4-grid 
configuration of the 0000 UTC RAMS forecast cycle, verified at METAR stations located on RAMS grid 4. 
Parameters that are plotted as a function of forecast hour include: a) mean observed temperatures (dashed) and 
mean forecast temperatures (solid), b) RMS error, c) bias, and d) error standard deviation (SD). The gaps that 
occur in the plot result from missing METAR observations for those times. 



















Figure 4.6. A meteogram plot of temperature errors (°C) for the 1999 warm season months for the 4-grid 
configuration of the 0000 UTC RAMS forecast cycle, verified at METAR stations located on RAMS grid 1. 
Parameters that are plotted as a function of forecast hour include: a) mean observed temperatures (dashed) and 
mean forecast temperatures (solid), b) RMS error, c) bias, and d) error standard deviation (SD). The gaps that 
occur in the plot result from missing METAR observations for those times. 
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2. Wind direction and speed 


The 0000 UTC RAMS forecast wind speed and direction errors for the KSC/CCAFS wind towers at the 
16.5-m level are shown in Figures 4.7 and 4.8. Both the forecast wind speed and direction are only slightly 
biased throughout the 24-h forecast. The mean wind speeds are slightly stronger than the observed values 
primarily during the daytime hours (forecast hours 15-24, Fig 4.7a). The maximum RMS error slightly exceeds 
2 m s' 1 between 17-h and 24-h (Fig. 4.7b). The total model error is composed of a 1 m s 1 bias (Fig. 4.7c) and a 
2 m s" 1 SD (Fig. 4.7d). The ECSB typically propagates westward through the KSC/CCAFS tower network 
during the late morning and early afternoon hours (1500-1800 UTC, Cetola 1997) corresponding to the 15-18- 
h forecasts from the 0000 UTC RAMS run. Therefore, these results suggest that the forecast wind speeds are 
slightly too strong following the passage of the ECSB. This statement is supported by the easterly (negative) u- 
wind bias shown in Figure B4 of Appendix B. 

The RMS error for wind direction from the 0-h RAMS forecast is less than 20° (Fig. 4.8a) whereas the 
magnitude of the bias at this time is less than 5° (Fig. 4.8b). In addition, the 0-h SD in wind direction is nearly 
the same magnitude as the RMS error (not shown) suggesting that the total error is due to non-systematic 
variability in the forecasts and/or observations. A previous study using data from the NASA Shuttle Landing 
Facility wind tower showed that the SD in observed wind direction is inversely proportional to the square root of 
wind speed (Merceret 1995). The empirical relationship for observed wind direction SD (cr^) found in his 
study is 


G WD — 


44 

4ws ’ 


(4.1) 


where WS is the wind speed in knots and WD is wind direction in degrees. Based on this formulation, the SD in 
observed wind direction is estimated to be 16-22° using the average observed KSC/CCAFS tower wind speeds 
from Figure 4.7a. Therefore, nearly all of the 0-h RMS error in wind direction from RAMS is likely due to the 
expected variance in the tower observations. 

The RMS error in wind direction grows substantially after initialization increasing from less than 20° at 
initialization to about 50° by the 3-h forecast (Fig. 4.8a). The maximum RMS error occurs during the nocturnal 
and early morning hours peaking at about 60—65°. The RMS error decreases after the 18-h forecast, generally 
ranging from 40—55° during the remaining 6 hours. The total model error in wind direction is almost entirely 
composed of non-systematic variability since the error SD is much larger in magnitude than the bias (not 
shown). It is important to note that approximately 16—22° of the 40—65° RMS error in wind direction is likely 
due to the variability in the tower observations. After accounting for variance in observations, the remaining 
RMS error in wind direction is on the order of 25—50°. These errors may result from the model’s inability to 
resolve explicitly small-scale turbulent eddies, especially those associated with light and variable wind regimes 
during the nocturnal and early morning hours. In addition, a more sophisticated initialization scheme, such as 
four-dimensional data assimilation, may reduce the growth rate of these wind direction errors by better 
incorporating the high-resolution observational data available in east-central Florida. 

The AMU also computed the wind direction errors for observed wind speeds > 1.5 m s' 1 in order to filter 
out errors associated with light and variable winds. Figure 4.9 shows the resulting RMS error and bias in wind 
direction using this threshold. The bias curve changes very little from Figure 4.8b. However, the RMS error 
decreases by as much as 15°, especially in the 3-12-h forecasts (0300 UTC to 1200 UTC) when the light wind 
regime dominates during the nocturnal and early morning hours. Nearly all of the decrease in the RMS error 
plot is associated with the decrease in the error SD (not shown). Thus, with wind speeds of at least 1.5ms , the 
maximum RMS error in the forecast wind direction is generally 45-55°, again primarily composed of non- 
systematic error. 

Similar patterns and magnitudes in wind speed and wind direction errors occur in the 1200 UTC RAMS 
cycle except that they are phase-shifted by 12 h (not shown). 
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Figure 4.7. A meteogram plot of wind speed errors (m s' 1 ) for the 1999 warm season months for the 4-grid 
configuration of the 0000 UTC RAMS forecast cycle, verified at the 16.5-m level of the KSC/CCAFS wind- 
tower network. Parameters that are plotted as a function of forecast hour include: a) mean observed wind speed 
(dashed) and mean forecast wind speed (solid), b) RMS error, c) bias, and d) error standard deviation (SD). 
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Figure 4.8. A meteogram plot of wind direction errors (degrees) for the 1999 warm season months for the 
4-grid configuration of the 0000 UTC RAMS forecast cycle, verified at the 16.5-m level of the KSC/CCAFS 
wind towers. Parameters that are plotted as a function of forecast hour include: a) RMS error and b) bias. 
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Figure 4.9. A meteogram plot of wind direction errors (degrees) associated with a 1.5-ms' 1 minim um 
wind speed threshold. The results are shown for the 4-grid configuration of the 0000 UTC RAMS forecast cycle 
and verified at the 16.5-m level of the KSC/CCAFS wind towers. Parameters that are plotted as a function of 
forecast hour include: a) RMS error and b) bias. 
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b. Upper Air (0000 UTC cycle , XMR) 


This section presents the 0000 UTC cycle verification at the XMR rawinsonde site. The times that contain 
the most data for verification at the XMR site are 1100, 1600, and 2200 UTC, (11, 16, and 22-h forecasts, 
respectively). The 1 1-h and 22-h forecasts at XMR are discussed in order to verify the 0000 UTC forecast cycle 
during the early morning and late afternoon regimes of the 1999 warm season. Time-height cross sections 
illustrate the evolution of wind errors at the 50-MHz DRWP at every forecast hour. 

1. Temperature and dew point 

During the 1999 warm-season, RAMS consistently generated a forecast temperature profile that was too 
stable, particularly during the daylight hours. The 0000 UTC cycle temperature errors for the 11-h and 22-h 
forecasts are shown in Figures 4.10 and 4.11, respectively. Below 650 mb the mean 1 1-h forecast temperatures 
are typically I-2°C cooler than the mean observed profile. Meanwhile, above 650 mb the mean forecast 
temperatures are slightly warmer than observed (Fig. 4.10a and c). The RMS error is largest at the surface 
(2.5°C) and decreases with height to about 1°C at 350 mb (Fig. 4.10b). The total model error represented by the 
RMS error is composed of a 1-2 °C negative bias below 650 mb and up to a 0.5°C warm bias above 650 mb 
(Fig. 4.10c). The bias in the lower troposphere is generally negative except at the surface, where a +1°C bias 
occurs during the early morning sounding. The non- systematic error represented by the SD decreases from 2 °C 
at the surface to about 1°C above 900 mb (Fig. 4.10d). 

At 22 h, the difference between the mean observed and forecast temperature profile increases, especially 
below 800 mb (Fig. 4.1 la). The total model error is at least 2°C from the surface to 900 mb (Fig. 4.1 lb) and the 
greatest increase in model error over the 1 1-h forecast occurs between 1000-900 mb. The cool bias in the lower 
troposphere extends down to the surface (Fig. 4.1 lc) and the error SD decreases by about 1°C at the surface, but 
remains approximately the same at upper levels (Fig. 4.1 Id). The warm bias above 650 mb increases in 
magnitude slightly compared to the 11-h forecast. Based on these results, RAMS demonstrates a tendency to 
develop and maintain a cool bias below 650 mb, especially at the surface during the daylight hours. 
Hydrostatically, this cool lower tropospheric temperature bias leads to a positive pressure bias with a magnitude 
approaching 3 mb near the surface (not shown). The surface-based cool bias does not appear during the 
nighttime hours, but exists at 22 hours. The total effect of the temperature bias is to produce a temperature 
profile that is too stable compared to observations. 

The vertical profile of dew point errors using data from the XMR rawinsonde site follows a different pattern 
than the temperature errors. Dew point errors tend to increase with height reaching a maximum at about 700 
mb, contrary to the temperature errors. The mean 1 1-h forecast dew point is similar to the mean observed dew 
point (Fig. 4.12a) as indicated by the small bias throughout the troposphere (magnitude < 1°C, Fig. 4.12c). The 
RMS error and SD traces are nearly identical and increase from 2°C near the surface to greater than 4°C in the 
mid-troposphere (Figs. 4.12b and d). The dew point errors decrease in magnitude above 600 mb. A similar 
pattern occurs in the dew point errors for the 22-h forecast, but with a slight dry bias at low levels and a slightly 
higher mid-tropospheric maximum error (see Fig. B5 in Appendix B). 
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Figure 4.10. Vertical profiles of temperature errors (°C) at XMR for the 1 1-h forecast from the 0000 UTC 
RAMS forecast cycle. Parameters are plotted as a function of pressure and include: a) mean observed 
temperatures (dashed) and mean forecast temperatures (solid), b) RMS error, c) bias, and d) error standard 
deviation (SD). 
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0000 UTC 4-grid Cycle 22-h Temp (C) at XMR 



Figure 4.1 1. Vertical profiles of temperature errors (°C) at XMR for the 22-h forecast from the 0000 UTC 
RAMS forecast cycle. Parameters are plotted as a function of pressure and include: a) mean observed 
temperatures (dashed) and mean forecast temperatures (solid), b) RMS error, c) bias, and d) error standard 
deviation (SD). 
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2. Wind direction and speed 


The 0000 UTC forecast wind directions at 11-h and 22-h both exhibit a large amount of non-systematic 
variability when compared to the bias. The greatest errors in wind direction occur in the lower troposphere 
below 800 mb at both times, but the errors have different structures. During the early morning (11-h forecast), 
the largest wind direction RMS error is found at the surface and is composed of primarily non-systematic 
variability (Fig. 4.13a). The forecast wind direction is virtually unbiased since the magnitude of the bias is 
typically much less than the RMS error throughout the troposphere (Fig. 4.13b). 

At 22 h, the largest RMS error (-65-70°) occurs at 850 mb (Fig. 4.14a), contrary to 1 1 hours. Once again, 
the RMS error is primarily composed of the non-systematic variability since the magnitude of the bias is much 
less than the RMS error (Fig. 4. 14b). However, there is a notable negative wind direction bias (-20°) above 850 
mb. As a simple geometric interpretation, a negative (positive) bias in wind direction indicates that the forecast 
wind direction is consistently counterclockwise/backed (clockwise/veered) from the observed wind. Based on 
an examination of the mean upper-level observed and forecast u- and v-wind components and their respective 
errors (not shown), the negative wind direction bias indicates that the forecast winds are too southerly relative to 
the observed westerly wind direction. The wind direction errors in the 1200 UTC forecast cycle are quite 
similar at the same valid time as discussed above, but with a slightly larger magnitude near 850 mb (not shown). 

The 0000 UTC cycle wind speed verification at the 1 1-h forecast is given in Figure 4.15. Near the surface, 
the mean forecast wind speeds are slightly stronger than the observed wind speeds whereas the contrary is true 
above 900 mb (Fig. 4.15a). The RMS error resembles the SD suggesting that non-systematic variability 
composes most of the model error (Figs. 4.15b and d). The smallest SD and RMS errors are found near the 
surface (2 m s* 1 ) and the magnitudes increase with height to about 4 m s" 1 at 300 mb. This result is consistent 
with previous findings in that the magnitude of wind-speed errors increases with higher observed wind speeds 
(Manobianco et al. 1996, Nutter and Manobianco 1999). The largest bias occurs at the surface (+1.5 m s" ) and 
only a slight negative bias (between 0 and -1 m s* 1 ) is evident above 900 mb (Fig. 4.15c). Very similar error 
patterns occur in the 22-h plots except that the magnitudes of the low-level positive and upper-level negative 
biases increase by as much as 1 ms' 1 (not shown). 


28 


Pressure (mb) Pressure (mb) 


0000 UTC 4-grid Cycle 1 1-h Dew Point (C) at XMR 


Observed 

Forecast 



-10 0 10 
Mean 



S 500 



8 600 



2 4 

RMS Error 


Figure 4.12. Vertical profiles of dew point errors (°C) at XMR for the 1 1-h forecast from the 0000 UTC 
RAMS forecast cycle. Parameters are plotted as a function of pressure and include: a) mean observed dew point 
(dashed) and mean forecast dew point (solid), b) RMS error, c) bias, and d) error standard deviation (SD). 








0000 UTC 4-grid Cycle 1 1-h Wind Dir (deg) at XMR 



Figure 4.13. Vertical profiles of wind direction errors (degrees) at XMR for the 1 1-h forecast from the 
0000 UTC RAMS forecast cycle. Parameters are plotted as a function of pressure and include: a) RMS error 
and b) bias. 


0000 UTC 4-grid Cycle 22-h Wind Dir (deg) at XMR 



Figure 4. 14. Vertical profiles of wind direction errors (degrees) at XMR for the 22-h forecast from the 
0000 UTC RAMS forecast cycle. Parameters are plotted as a function of pressure and include: a) RMS error 
and b) bias. 
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0000 UTC 4-grid Cycle 1 1-h Wind Spd (m/s) at XMR 
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Figure 4.15. Vertical profiles of wind speed errors (m s 1 ) at XMR for the 1 1-h forecast from the 0000 
UTC RAMS forecast cycle. Parameters are plotted as a function of pressure and include: a) mean observed 
wind speed (dashed) and mean forecast wind speed (solid), b) RMS error, c) bias, and d) error standard 
deviation (SD). 






c. Upper Air (0000 UTC cycle at 50 MHz DRWP) 


The interpolation of RAMS forecast upper-level winds to each gate of the KSC/CCAFS 50-MHz DRWP 
provides the best continuity in verifying the model wind field above 2 km. A significant amount of 
observational data was available at each forecast hour to provide a seasonally representative verification (as 
discussed in Section 3). Figure 4.16 shows the verification of 0000 UTC cycle wind direction as a function of 
forecast hour and height for the 24-h model integration. The largest RMS error in wind direction occurs 
between 2-6 km for nearly all forecast hours, ranging from 30-60° (Fig. 4.16a). The smallest errors are found 
at upper levels above 8 km (20-30° given by the lightest shading in Fig. 4.16a) mainly during the first 12 h of 
the integration. The bias in wind direction is generally less than 10° at all times and levels suggesting that 
RAMS forecast wind direction at upper levels is unbiased (Fig. 4.16b). Again, these wind direction errors may 
be reduced by using a more advanced initialization scheme such as four-dimensional data assimilation. 

Figure 4.17 shows the verification of the 0000 UTC cycle wind speed as a function of forecast hour and 
height for the 24-h model integration. The primary feature to note in the verification of upper-level wind speeds 
is that RAMS tends to under forecast the magnitude of the wind slightly in the 10-14-km layer after 6 h (refer to 
bias in Fig. 4.17d). The mean observed and forecast wind speeds are generally similar except that the mean 
10-14-km forecast wind speed decreases as a function of forecast hour, particularly after 6 h (Fig. 4.17b). As a 
result, the wind speed RMS error increases slightly with time in the 10-14-km layer (Fig. 4.17c). RAMS 
forecasts generate up to a 2-m s" 1 positive wind speed bias in this layer during the first few hours of integration 
followed by a negative bias up to 3 m s' 1 after 6 h (Fig. 4.17d). A 1-2 m s' 1 negative wind speed bias also 
develops at lower levels after the 18-h forecast time. For further reference, the 1200 UTC cycle wind direction 
and speed errors at the 50-MHz DRWP site are shown in Appendix B, Figures B6 and B7. 
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Figure 4.16. Time-height cross sections of wind direction errors at the KSC/CCAFS 50-MHz DRWP for 
the 0000 UTC RAMS forecast cycle. Parameters are contoured for every hour and include: a) RMS error 
(shaded every 10°), and b) bias (contoured at ± 5°, 10°, 20°, and 30°, with negative values less than -5° shaded). 
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Figure 4.17. Time-height cross sections of wind speed errors at the KSC/CCAFS 50-MHz DRWP for the 
0000 UTC RAMS forecast cycle. Parameters are contoured for every hour and include: a) mean observed wind 
speed (shaded every 2 m s’ 1 ), b) mean forecast wind speed (shaded every 2 m s’ 1 ), c) RMS error (shaded every 1 
m s’ ), and d) bias (negative values shaded every 1 m s’ 1 ). 
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4.1.2 Four-Grid/Three-Grid Configuration Comparison 

The purpose of this section is to illustrate the effect of a decrease in horizontal resolution of the inner most 
RAMS grid on the subsequent model errors. The 3-grid configuration represents the coarser solution in which 
grid 4 is withheld from the RAMS forecasts. 

4.L2.1 Summary of Surface (0000 UTC forecast cycles at KSC/CCAFS towers) 


Table 4.4. Summary of the 4-grid/3-grid RAMS error comparison at the KSC/CCAFS wind towers 
for the 0000 UTC forecast cycle. Temperature and dew point errors are valid at 1.8 m whereas wind 

| direction and speed errors are valid at 16.5 m. 



Variable 

Config. 

RMS Error 

Bias 

Notable Errors f 

Temp 

(°C) 

4-grid 

3-grid 

1 to 5 

-4 to 1 

• 

The 3-grid configuration of RAMS has a more 
pronounced cool bias during the day. 

1 to 9 

-7 to 1 

• 

Errors are nearly identical at night. 

Dew 

Point 

(°C) 

4-grid 

1 to 3 

-2 to 0 

• 

The 3-grid configuration of RAMS has a more 
pronounced dry bias during the day. 

3 -grid 

1 to 9 

-7 to 0 

• 

The 3-grid dry bias begins after 3 h, during the 
nocturnal hours. 

Wind 

Direction 

(deg) 




• 

Both forecasts are virtually unbiased and have 

4-grid 

15 to 60 

-10 to 5 


about a 60° RMS error after the 6-h forecast. 

3-grid 

15 to 65 

-15 to 5 

• 

RMS error is slightly smaller in the 4-grid 
forecasts when comparing model runs that did 
not generate significant precipitation. 


Wind 
Speed 
(m s' 1 ) 

4-grid 

1.5 to 2.5 

0 to 1.5 

• 

• 

Generally, very minor differences. 

The 3-grid forecast wind speed is unbiased 

3 -grid 

1.5 to 2.5 

0 to 0.5 


between 12-24 h whereas the 4-grid wind 
forecast has a+lm s* 1 bias. 
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4.1. 2.2 Summary of Upper Air (0000 UTC forecast cycle atXMR) 


Table 4.5. Summary of the 4-grid/3-grid RAMS error comparison at the XMR rawinsonde site. 

Variable 

Config. 

RMS Error 

Bias 

Notable Errors 

Temp 

(°C) 

4-grid 

3-grid 

1 to 3.2 
1 to 5 

-2.5 to 1 
-4 to 0.5 

• The low-level temperature errors (below 950 
mb) are much larger in the 3-grid forecasts. 

• The 3-grid configuration has a slightly smaller 
warm bias above 600 mb. 

Dew 

Point 

(°C) 

4-grid 

3-grid 

2 to 5.5 
1.5 to 5 

-1.5 to 1.5 
-2.8 to 2 

• The low-level dew point errors are much 
larger in the 3 -grid forecasts. 

• At upper-levels between 450-600 mb, the 3- 
grid forecasts exhibit a 1-2°C moist bias 
compared to a -1° to 0°C bias in the 4-grid 
forecasts. 

Wind 

Direction 

(deg.) 

4-grid 
3 -grid 

30 to 80 
35 to 70 

-20 to 15 
-30 to 5 

• The 3-grid configuration has about a 10° 
increase in the RMS error over the 4-grid 
configuration, near the surface, in the 750-550 
mb layer, and above 500 mb. 

Wind 
Speed 
(m s' 1 ) 

4-grid 

3-grid 

2.5 to 4 

1.5 to 4 

-1.5 to 2 
-1.5 to 0.5 

• Below 950 mb, the 4-grid RMS error is about 
1 m s* 1 greater than the 3-grid results. 

• The 4-grid RMS error is 0. 5-1.0 m s’ 1 less 
than the 3-grid error in the 600-800 mb layer. 

• Above 500 mb, the 3-grid bias is less negative 
than the 4-grid bias. 


4. 1. 2. 3 Detailed Discussion 

a . Surface (0000 UTC forecast cycle at KSC/CCAFS towers) 

1. Temperature and dew point 

The most substantial difference between the 4- and 3-grid RAMS forecasts is the magnitude of the near- 
surface temperature and dew point errors. These model errors are both larger in the 3-grid forecasts. For nearly 
all forecast hours, the differences in wind forecasts are negligible at the surface. The next several figures 
illustrate the differences and similarities in the error characteristics of the 4-grid and 3-grid RAMS 
configurations. The results are shown for the 1.8-m and 16.5-m levels of the KSC/CCAFS tower network. 

The results from the 0000 UTC cycle show that the 1 .8-m temperature errors at the KSC/CCAFS towers are 
larger in the 3-grid configuration, particularly during the daylight hours. Both mean forecast traces follow the 
mean observed temperature closely until the 1 1-h forecast (Fig. 4.18a). After 1 1 h, both mean forecast traces 
diverge substantially from the observed temperature. The 4-grid mean forecast temperature shows some diurnal 
increase after 1 1 h, however, the 3-grid mean forecast temperature barely increases above the early morning 
mean minimum temperature. As a result, the 3-grid RMS error in forecast temperature is substantially larger 
than the 4-grid RMS error, peaking at nearly 9°C at 17 h compared to a 5°C maximum 4-grid RMS error at 19 h 
(Fig. 4.18b). The 3-grid cool bias exceeds -6°C between 16-19 h compared to a -3 to -4°C bias in the 4-grid 
configuration (Fig. 4.18c). In addition to a larger cool bias, the non-systematic error is also larger in the 3-grid 
configuration as shown in Figure 4. 18d. The 3-grid SD is about 2.5°C larger than the 4-grid SD from 15-18 h. 

The RAMS dew point errors at the 1.8-m level of the KSC/CCAFS towers are also larger in the 3-grid 
forecasts as depicted in Figure 4. 19. The 4-grid forecast dew point follows the observed dew point closely until 
12 h after which it remains about 2°C too low (Fig. 4.19a). However, the mean 3-grid forecast dew point drops 
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substantially after 3 h through the 17-h forecast. Thus, the mean 3-grid forecast dew point is over 7°C lower 
than the mean observed dew point (by the 17-h forecast). The 3-grid dew point RMS error approaches 9°C at its 
maximum (Fig. 4.19b) composed of a dry bias exceeding -6°C at 16-19 h (Fig. 4.19c) and a SD of 6°C 
preceding the time of the maximum bias (Fig. 4.19d). Each of these errors exceeds the 4-grid dew point errors 
by 2-4°C or more after the 3-h forecast. 

The 3-grid temperature errors in the 1200 UTC cycle are not as large as in the 0000 UTC cycle but the 3- 
grid errors still exceed the 4-grid temperature errors. However, the 3-grid dew point errors in the 1200 UTC 
cycle have similar error magnitudes as in the 0000 UTC cycle (see Appendix C, Figs. Cl -2). 

2. Wind direction and speed 

The 0000 UTC forecast cycle comparison between the 4- and 3 -grid wind direction and speed forecasts at 
the 16.5-m tower level is shown in Figures 4.20 and 4.21. Only a negligible difference occurs in the RMS error 
and bias errors for forecast wind direction (Figs. 4.20a-b). With the exception of forecast hours 21—24, the 3- 
grid RMS error is generally within 5° of the 4-grid error. The most substantial difference is the increase in the 
negative bias between 12-17 h; however, the -10° bias in the 3-grid configuration is still small compared to the 
magnitude of the RMS error. 

The likely cause for the deviation in wind direction errors during 21-24 h is that many 4-grid forecasts do 
not complete the 24 hour integration during periods of active convection in the model (as mentioned in Section 
3.2). As a result, the 4-grid sample size during 21-24 h consists of a greater percentage of forecasts in quiescent 
regimes when models errors in wind direction are likely to be smaller. The 3 -grid configuration does not 
experience the same run-time limitations as the 4-grid configuration. Thus, the 3-grid forecasts complete all 24 
hours of the integration and include both disturbed and quiescent regimes. In fact, the sample size of forecast 
hour 24 for the 3 -grid configuration is over twice that of the 4-grid sample size (not shown). Therefore, one 
would expect that the 3-grid wind direction errors should be larger during forecast hours 21-24. 

To address this discrepancy from 21-24 h, the 4- and 3-grid errors were recomputed only for 4-grid 
forecasts that completed all 24 hours. The wind direction RMS errors from this analysis indicate that both 
configurations follow a similar pattern in the error evolution and do not diverge dramatically after 21 h (Fig. 
4.20c). However, the 4-grid RMS errors in this comparison are 5-15° less than the 3-grid errors between 9-16 
h, and again after 21 h. In addition, the 4-grid wind direction is less biased than the 3-grid wind direction, 
particularly between 9-18 h (Fig. 4.20d). 

The comparison between the 4- and 3 -grid forecast wind speeds shows that only small differences occur 
between the two configurations. The mean 4- and 3-grid forecast wind speeds are nearly exact from 0-12 h, 
after which the mean 3 -grid wind speeds converge towards the mean observed wind speed whereas the mean 4- 
grid wind speeds do not (Fig. 4.21a). The RMS error and SD are also nearly identical throughout all 24 hours 
(Figs. 4.21b and d). The most significant difference between the 4-grid and 3-grid forecast wind speeds is that 
the 3-grid wind speed has virtually no bias after 12 h whereas the 4-grid configuration experiences about a +1- 
m s‘* bias after 12 h (Fig. 4.21c). By comparing the 4-grid forecasts that completed all 24 hours to the 3-grid 
forecasts at the same times, the larger wind speed bias in the 4-grid configuration remains (not shown). In the 
1200 UTC forecast cycle, the 4-grid/3-grid differences in wind direction and speed errors are smaller than the 
0000 UTC cycle (not shown). 


0000 UTC 3/4-grid Cycle Temperature (C) for 1.8-m towers 




Figure 4.18. A meteogram plot that displays a comparison between the 0000 UTC forecast cycle surface 
temperature errors (°C) from the 4- and 3-grid RAMS configurations. Surface temperatures are verified at the 
1.8-m level of the KSC/CCAFS wind tower network. Parameters plotted as a function of forecast hour for both 
the 4-grid and 3-grid RAMS configurations include: a) mean observed temperature, mean 4-grid forecast 
temperature, and mean 3-grid forecast temperature, b) RMS error, c) bias, and d) error standard deviation (SD). 
The plotting convention is a solid line for the 4-grid forecasts, dot-dashed line for the 3-grid forecasts, and a 
dashed line for observed values. 
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Figure 4.19. A meteogram plot that displays a comparison between the 0000 UTC forecast cycle surface 
dew point errors (°C) from the 4- and 3-grid RAMS configurations. Surface dew points are verified at the 1 .8-m 
level of the KSC/CCAFS wind tower network. Parameters plotted as a function of forecast hour include: a) 
mean observed dew point, mean 4-grid forecast dew point, and mean 3-grid forecast dew point, b) RMS error, 
c) bias, and d) error standard deviation (SD). The plotting convention is a solid line for the 4-grid forecasts, 
dot-dashed line for the 3-grid forecasts, and a dashed line for observed values. 
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0000 UTC 3/4-grid Cycle Wind Dir (deg) for 16.5-m towers 




0000 UTC 3/4-grid Cycle Wind Dir (deg) for 16.5-m towers 




Forecast Hour 


Figure 4.20. A meteogram plot that displays a comparison between the 0000 UTC forecast cycle near- 
surface wind direction errors (degrees) from the 4-grid and 3-grid RAMS configurations. Near-surface wind 
direction is verified at the 16.5-m level of the KSC/CCAFS wind tower network. Parameters plotted as a 
function of forecast hour for both the 4-grid and 3-grid RAMS configurations include: a) RMS error and b) bias 
for all available forecasts, and c) RMS error and d) bias only for forecasts when the 4-grid configuration 
completed all 24 hours. The plotting convention is a solid line for the 4-grid errors and a dot-dashed line for the 
3 -grid errors. 
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0000 UTC 3/4-grid Cycle Wind Spd (m/s) for 16.5-m towers 



0 3 6 9 12 15 18 21 24 



0 3 6 9 12 15 18 21 24 



0 3 6 9 12 15 18 21 24 

Forecast Hour 


Figure 4.21. A meteogram plot that displays a comparison between the 0000 UTC forecast cycle near- 
surface wind speed errors (m s' 1 ) from the 4- and 3-grid RAMS configurations. Near- surface wind speeds are 
verified at the 16.5-m level of the KSC/CCAFS wind tower network. Parameters plotted as a function of 
forecast hour for both the 4-grid and 3-grid RAMS configurations include: a) mean observed wind speed, mean 
4-grid forecast wind speed, and mean 3-grid forecast wind speed, b) RMS error, c) bias, and d) error standard 
deviation (SD). The plotting convention is a solid line for the 4-grid forecasts, dot-dashed line for the 3-grid 
forecasts, and a dashed line for observed values. 
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b. Upper Air (0000 UTC forecast cycle at XMR) 


In the upper-air forecasts, the 4-grid/3-grid runs exhibited generally small differences in the forecast errors 
for all variables. Only comparisons for the 22-h forecast from the 0000 UTC cycle (post sea breeze regime) at 
the XMR rawinsonde site are shown in this section. The 4-grid/3-grid plots at the 1 1-h forecast demonstrate 
similar characteristics to the plots shown in this section. Furthermore, the 4-grid/3-grid error differences are 
small in the verification data at the 50-MHz DRWP site (see Appendix C, Figs. C3-6). 

1. Temperature and dew point 

Figure 4.22 shows a vertical profile of both the 4-grid and 3-grid mean temperatures, RMS error, bias, and 
SD at XMR. With the exception of the surface, the mean forecast temperature profiles are very similar and have 
the same deviation from the mean observed temperature profile (Fig. 4.22a). Each of the temperature error 
profiles shown in Figures 4.22b-d also show that with the exception of the lowest 50 mb, little difference occurs 
between the 4-grid and 3-grid configuration of RAMS. Perhaps the most notable difference at upper levels is 
that the 3 -grid configuration exhibits a smaller warm bias above 650 mb compared to the 4-grid configuration 
(Fig. 4.22c). 

The 4-grid/3-grid profiles of dew point errors at XMR are given in Figure 4.23 for forecast hour 22. The 
most significant differences in these profiles occur in the lowest 50 mb and between 450 and 550 mb. In the 
lowest 50 mb, the 3-grid forecast dew point experiences an RMS error about 2°C greater than the 4-grid 
configuration (Fig. 4.23b), composed of an increase of about 1.5°C in the magnitude of the bias and SD (Figs. 
4.23c-d). In the 450-550 mb layer the 3-grid forecasts exhibit a +1 to +2°C bias in dew point compared to the 
virtually unbiased 4-grid forecast. Other than these differences, the 4- and 3-grid dew point forecasts at XMR 
are fairly similar. 
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0000 UTC 3/4-grid Cycle 22-h Temp (C) at XMR 



Mean Bias 



Figure 4.22. Vertical profiles of temperature errors (°C) at XMR for the 22-h forecast, displaying a 
comparison between the 4- and 3-grid configurations of RAMS from the 0000 UTC forecast cycle. Parameters 
plotted as a function of pressure for both the 4-grid and 3 -grid RAMS configurations include: a) mean observed 
temperature, mean 4-grid forecast temperature, and mean 3-grid forecast temperature, b) RMS error, c) bias, and 
d) error standard deviation (SD). The plotting convention is a solid line for the 4-grid forecasts, dot-dashed line 
for the 3-grid forecasts, and a dashed line for observed values. 
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0000 UTC 3/4-grid Cycle 22-h Dew Point (C) at XMR 



02468 02468 

RMS Error SD 


Figure 4.23. Vertical profiles of dew point errors (°C) at XMR for the 22-h forecast, displaying a 
comparison between the 4- and 3-grid configurations of RAMS from the 0000 UTC forecast cycle. Parameters 
plotted as a function of pressure for both the 4-grid and 3-grid RAMS configurations include: a) mean observed 
dew point, mean 4-grid forecast dew point, and mean 3 -grid forecast dew point, b) RMS error, c) bias, and d) 
error standard deviation (SD). The plotting convention is a solid line for the 4-grid forecasts, dot-dashed line 
for the 3-grid forecasts, and a dashed line for observed values. 
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2. Wind direction and speed 


The 4-grid versus 3 -grid forecast wind direction errors at XMR also show a fair amount of similarity. The 
RMS error is typically 10° greater in the 3-grid forecasts near the surface and in the 750-400 mb layer (Fig. 
4.24a) and is composed of primarily non-systematic variability given by the small bias in Figure 4.24b. The 
differences in the bias are not substantial since the magnitudes of the bias are much less than the RMS errors 
(Fig. 4.24b). 

Finally, the 3-grid configuration shows some small improvements in the wind speed bias over the 4-grid 
configuration at 22 hours, but the 3-grid RMS error is still similar to the 4-grid forecasts. Figure 4.25 shows the 
mean forecast and observed wind speed profiles at XMR along with the RMS error, bias, and SD. From the 
surface to 800 mb, the 4-grid forecasts have a positive wind speed bias up to 2 m s‘ l whereas the 3-grid bias is 
generally less than 0.5 m s‘ l (Fig. 4.25c). This decrease in bias leads to an improvement in the RMS error of the 
3-grid results relative to the 4-grid error, mainly below 900 mb (Fig. 4.25b). At levels above 500 mb, the 3-grid 
forecasts experience a smaller negative wind speed bias compared to the 4-grid forecasts. However, the total 
RMS error in the 3-grid configuration is typically greater than or equal to the 4-grid RMS error from 900—400 
mb due to the larger SD in the 3-grid forecasts, particular between 600 and 800 mb (Fig. 4.25d). 


0000 UTC 3/4-grid Cycle 22-h Wind Dir (deg) at XMR 



RMS Error Bias 


Figure 4.24. Vertical profiles of wind direction errors (degrees) at XMR for the 22-h forecast, displaying a 
comparison between the 3-grid and 4-grid configurations of RAMS from the 0000 UTC forecast cycle. 
Parameters plotted as a function of pressure for both the 4-grid and 3-grid RAMS configurations include: a) 
RMS error and b) bias. The plotting convention is a solid line for the 4-grid configuration errors and a dot- 
dashed line for the 3-grid configuration errors. 
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0000 UTC 3/4-grid Cycle 22-h Wind Spd (m/s) at XMR 




Figure 4.25. Vertical profiles of wind speed errors (m s' 1 ) at XMR for the 22-h forecast displaying a 
comparison between the 4- and 3-grid configurations of RAMS from the 0000 UTC forecast cycle. Parameters 
plotted as a function of pressure for both the 4 -grid and 3 -grid RAMS configurations include: a) mean observed 
wind speed, mean 4-grid forecast wind speed, and mean 3-grid forecast wind speed, b) RMS error, c) bias, and 
d) error standard deviation (SD). The plotting convention is a solid line for the 4-grid forecasts, dot-dashed line 
for the 3 -grid forecasts, and a dashed line for observed values. 
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4.1.3 Eta Benchmark Results 


The AMU performed a benchmark comparison between the RAMS 4-grid configuration and Eta point 
forecasts at 14 stations across the southeastern United States (refer to Fig. 2.1 and Table 2.1). This section 
provides a comparison between the 1200 UTC cycle RAMS 4-grid configuration and Eta point forecasts at TTS, 
which is the only Eta point forecast available on grid 4. The verification at Orlando, FL (MCO, on RAMS grid 
3) is given in Appendix D. The differences between the RAMS and the Eta model errors on grids 1 and 2 are 
generally insignificant and not discussed in this paper. Refer to Figure 2.1 for the locations of each of these 
stations used in the Eta/RAMS comparison. 

In addition to surface forecasts, the verification of both the RAMS 4-grid configuration and the Eta point 
forecasts at mandatory pressure levels is shown for a composite of four Florida rawinsonde sites [Miami 
(72202), Jacksonville (72206), Tampa Bay (72210), and CCAFS (74794)]. However, veiy few soundings from 
74794 were available in the mandatory pressure data set from MIDDS. Thus, the majority of the upper-air 
comparison is valid for 72202, 72206, and 72210. 

4.L3A Summary of Surface (1200 UTC cycle at TTS) 


Table 4.6. 

Summary of the 4-grid RAMS and Eta model error comparison conducted at TTS for the 
1200 UTC forecast cycle. 

Variable 

Model 

RMS Error 

Bias 

Notable Errors 


RAMS 

1 to 4 

-3 to 2 

• The Eta model outperforms RAMS during the 

Temp 




daytime hours evidenced by a 2-3°C cool bias 

(°C) 

Eta 

1 to 2 

-0.5 to 2 

in RAMS and virtually no bias in the Eta. 



1 to 2 

Oto 1.5 

• After RAMS initialization, the Eta model has a 





1°C moist bias whereas RAMS is unbiased. 



1.5 to 2 

1 to 1.5 


Wind 

RAMS 

30 to 75 

0 to 20 

• During most forecast hours, the Eta model has 

Direction 




a larger positive bias than RAMS. 

(deg.) 

Eta 

40 to 65 

5 to 30 






• RAMS has approximately a 1 ms* 1 smaller 





RMS error than the Eta model, particularly 

Wind 

RAMS 

1 to 2 

-1 to 0.5 

during the nocturnal hours. 

(m s' 1 ) 

Eta 

1 to 2 

-0.5 to 2 

• A 1 .5 m s' 1 bias occurs in the Eta model during 





the nocturnal hours compared to virtually no 





bias in RAMS. 
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4.1. 3.2 Summary of Upper Air (1200 UTC cycle at rawinsondes over the FL peninsula) 


Table 4.7. Summary of the 4-grid RAMS and Eta model comparison conducted at the rawinsondes 
in the Florida peninsula (mandatory pressure levels only) for the 1200 UTC forecast cycle. The 12-h 
forecast valid at 0000 UTC is presented in this report. 

Variable 

Model 

RMS Error 

Bias 

Notable Errors 

Temp 

PC) 

RAMS 

Eta 

1 to 2.5 
1 to 2 

-1.5 to 0.5 
-0.5 to 0.5 

• Both RAMS and the Eta model generate 
temperature profiles that are too stable relative 
to observations. 

• RAMS has a low-level cool bias about 1°C 
greater than the Eta model. 

• Both models experience about a +1°C bias 
above 550 mb. 

Dew 

Point 

PC) 

RAMS 

Eta 

1 to 4 
0.5 to 4 

-0.2 to 0.5 
-2 to 1.5 

• The RAMS and the Eta model have a very 
similar RMS error, reaching a maximum of 
4°C at 700 mb. 

• The Eta model exhibits a low-level moist bias 
of 1.5°C and a dry bias of 1-2°C between 
850-500 mb whereas RAMS dew point 
forecasts are generally unbiased. 

Wind 

Direction 

(deg.) 

RAMS 

Eta 

30 to 60 
30 to 50 

-10 to 10 
Oto 10 

• Both models are virtually unbiased from the 
surface to 300 mb. 

• The RMS error in the Eta model is about 
10-15° less than RAMS at several levels. 

Wind 
Speed 
(m s' 1 ) 

RAMS 

Eta 

2.5 to 3.5 
2.2 to 3.5 

-0.5 to 1.5 
-0.5 to 0 

• The RAMS and Eta model errors are very 
similar throughout the troposphere. 

• Near the surface, RAMS has a +1.5-2.0 m s* 1 
bias whereas the Eta is virtually unbiased. 


4. 1. 3. 3 Detailed Discussion 
a. Surface (1200 UTC cycle at TTS) 

1. Temperature and dew point 

For surface temperature forecasts from the 1200 UTC cycle, the Eta model outperforms RAMS during the 
daylight hours. The mean Eta forecast temperature follows the mean observed temperature fairly closely during 
the 24-h period whereas the mean RAMS forecast temperature deviates by 2-3°C primarily during 6-15 h (Fig. 
4.26a). The Eta model temperature RMS error is generally 2°C during the entire 24-h forecast period composed 
primarily of non-systematic error and little bias (Figs. 4.26b-d). Meanwhile, the RAMS RMS error is 
substantially larger than the Eta model during the daytime with RMS error approaching 4°C (Fig. 4.26b). As 
discussed in the previous sections, the RAMS temperature RMS error consists of a -2 to -3°C cool bias in 
addition to a non-systematic error similar to that of the Eta model (Figs. 4.26c-d). 

Contrary to the temperature predictions, the RAMS performance is nearly equivalent to the Eta model in 
dew point forecasts at TTS. The mean RAMS dew point forecasts follow the mean observed dew point trace 
more closely than the Eta model as shown in Figure 4.27a. However, the RMS error in both models is virtually 
the same despite RAMS being less biased than the Eta model (Figs. 4.27b-c). For a version of the Eta model 
run at 29 km resolution, Nutter and Manobianco (1999) identified a slight moist bias at TTS. In this study, the 
Eta model also exhibits a moist bias at TTS (~ 1°C), consistent with results found in Nutter and Manobianco. 
The total error for RAMS is composed of a slightly higher error SD, particularly between 8-1 1 h (Fig. 4.27d). 
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2. Wind direction and speed 


With a few exceptions, the two models perform similarly in the wind direction and speed forecasts at TTS 
shown in Figures 4.28 and 4.29, respectively. The Eta model has a larger positive wind direction bias compared 
to RAMS, particularly during the afternoon and nighttime hours (Fig. 4.28b). However, RAMS experiences at 
least as much non-systematic variability as the Eta model (not shown), thus the total RMS error in both models 
is comparable, generally 40-60° for most forecast hours (Fig. 4.28a). The initial wind direction RMS error in 
RAMS is smaller than the Eta model because of the ingestion of local mesoscale data sets using ISAN, but the 
error quickly grows by 2 h. As shown in Figure 4.29, the wind speed forecasts are similar between 0 and 12 h; 
however, after 12 h the Eta model develops a +1.5 m s' 1 bias whereas RAMS is virtually unbiased. Because of 
this nocturnal bias at TTS, the RAMS RMS error is smaller than the Eta model between 12-24 h (Fig. 4.29b). 



0 3 6 9 12 15 18 21 24 



0 3 6 9 12 15 18 21 24 



Figure 4.26. A meteogram plot that displays a comparison between the 1200 UTC forecast cycle surface 
temperature errors (°C) from the RAMS 4-grid configuration and the Eta model. Surface temperatures are 
verified at TTS since this is the only station on grid 4 where Eta point forecasts are available. Parameters 
plotted as a function of forecast hour for both RAMS and the Eta model include: a) mean observed temperature, 
mean RAMS forecast temperature, and mean Eta forecast temperature, b) RMS error, c) bias, and d) error 
standard deviation (SD). The plotting convention is a solid line for the RAMS 4-grid forecasts, dot-dashed line 
for the Eta model, and a dashed line for observed values. 
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1200 UTC Eta/4-grid Cycle Dew point (C) for TTS 
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Figure 4.27. A meteogram plot that displays a comparison between the 1200 UTC forecast cycle surface 
dew point errors (°C) from the RAMS 4-grid configuration and the Eta model. Surface dew points are verified 
at TTS since this is the only station on grid 4 where Eta point forecasts are available. Parameters plotted as a 
function of forecast hour for both RAMS and the Eta model include: a) mean observed dew point, mean RAMS 
forecast dew point, and mean Eta forecast dew point, b) RMS error, c) bias, and d) error standard deviation 
(SD). The plotting convention is a solid line for the RAMS 4-grid forecasts, dot-dashed line for the Eta model, 
and a dashed line for observed values. 
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1200 UTC Eta/4-grid Cycle Wind Dir (deg) for TTS 



Forecast Hour 


Figure 4.28. A meteogram plot that displays a comparison between the 1200 UTC forecast cycle surface 
wind direction errors (degrees) from the RAMS 4-grid configuration and the Eta model. Surface wind direction 
is verified at TTS since this is the only station on grid 4 where Eta point forecasts are available. Parameters 
plotted as a function of forecast hour for both RAMS and the Eta model include: a) RMS error and b) bias. The 
plotting convention is a solid line for the RAMS 4-grid errors and a dot-dashed line for the Eta model errors. 
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1200 UTC Eta/4-grid Cycle Wind Spd (m/s) for TTS 



Forecast Hour 


Figure 4.29. A meteogram plot that displays a comparison between the 1200 UTC forecast cycle surface 
wind speed errors (m s" 1 ) from the RAMS 4-grid configuration and the Eta model. Surface wind speed is 
verified at TTS since this is the only station on grid 4 where Eta point forecasts are available. Parameters 
plotted as a function of forecast hour for both RAMS and the Eta model include: a) mean observed wind speed, 
mean RAMS forecast wind speed, and mean Eta forecast wind speed, b) RMS error, c) bias, and d) error 
standard deviation (SD). The plotting convention is a solid line for the RAMS 4-grid forecasts, dot-dashed line 
for the Eta model, and a dashed line for observed values. 
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b. Upper Air ( 1 200 UTC cycle at rawinsondes over the FL peninsula) 

1. Temperature and dew point 

Both RAMS and the Eta model generate a vertical temperature profile that is too stable relative to the 
observed temperature profile. The temperature errors plotted at mandatory pressure levels are shown in Figure 
4.30 for the 12-h forecast from the 1200 UTC cycle. Both the RAMS and Eta model have the largest RMS error 
below 900 mb with RAMS experiencing an RMS error that is 0.5°C greater than the Eta model (Fig. 4.30b). 
The larger RMS error is primarily caused by a lower-tropospheric cool bias that is 1°C larger than the Eta model 
(Fig. 4.30c). 

The total model error in dew point temperature between RAMS and the Eta model are quite similar 
throughout the troposphere as shown in Figure 4.3 1 . Both the RMS error and SD plots depict a nearly identical 
error profile in the RAMS and Eta model (Figs. 4.31b and d). The most substantial difference occurs near the 
surface and between 600-800 mb. At these levels, the Eta model RMS error is slightly larger than RAMS due 
to a 1.5°C moist bias near the surface and a 1-2°C dry bias between 500-800 mb (Fig. 4.31c). 

2. Wind direction and speed 

Figure 4.32 depicts a comparison between RAMS and the Eta model for the mean and error plots in wind 
direction for the 12-h forecast. Based on these profiles, the total model error for the Eta model is about 10-15° 
less than RAMS at several mandatory pressure levels up to 500 mb (Fig. 4.32a). Both models are virtually 
unbiased because the magnitudes of the biases are much less than the magnitudes of the RMS error (Fig. 4.32b). 
The increase in wind direction RMS error in RAMS is a result of an increase in non-systematic variability given 
by the larger error SD relative to the Eta model (not shown). 

Finally, the profiles of mean and error quantities of wind speed for the RAMS and the Eta model are given 
in Figure 4.33. These error profiles are nearly identical with an exception at 1000 mb. At 1000 mb, RAMS 
exhibits a positive wind speed bias of nearly 2 m s' 1 as opposed to the virtually unbiased Eta model (Fig. 4.33c). 
As a result, the RMS error in wind speed for RAMS (3 m s" 1 ) is about 1 m s' 1 larger than the Eta model (2 m s' 1 ). 
The wind speed errors of the Eta model are consistent with the results presented in Nutter and Manobianco 
(1999) for XMR and Tampa Bay, FL (TBW). 
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300 


1200 UTC Eta/4-grid Cycle 12-h Temp (C) at NATL 



Figure 4.30. Vertical profiles of temperature errors (°C) at the national rawinsondes on the Florida 
peninsula for the 12-h forecast, displaying a comparison between the 4-grid configuration of RAMS and the Eta 
model from the 1200 UTC forecast cycle. Parameters plotted as a function of pressure (mandatory levels only) 
for both RAMS and the Eta model include: a) mean observed temperature, mean RAMS forecast temperature, 
and mean Eta forecast temperature, b) RMS error, c) bias, and d) error standard deviation (SD). The plotting 
convention is a solid line for the RAMS 4-grid forecasts, dot-dashed line for the Eta model, and a dashed line 
for observed values. 
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1200 UTC Eta/4-grid Cycle 12-h Dew Point (C) at NATL 



Figure 4.3 1 . Vertical profiles of dew point errors (°C) at the national rawinsondes on the Florida peninsula 
for the 12-h forecast, displaying a comparison between the 4-grid configuration of RAMS and the Eta model 
from the 1200 UTC forecast cycle. Parameters plotted as a function of pressure (mandatory levels only) for 
both RAMS and the Eta model include: a) mean observ ed dew point, mean RAMS forecast dew point, and mean 
Eta forecast dew point, b) RMS error, c) bias, and d) error standard deviation (SD). The plotting convention is a 
solid line for the RAMS 4-grid forecasts, dot-dashed line for the Eta model, and a dashed line for observed 
values. 
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1200 UTC Eta/4-grid Cycle 12-h Wind Dir (deg) at NATL 




RMS Error Bias 


Figure 4.32. Vertical profiles of wind direction errors (degrees) at the national rawinsondes on the Florida 
peninsula for the 12-h forecast, displaying a comparison between the 4-grid configuration of RAMS and the Eta 
model from the 1200 UTC forecast cycle. Parameters plotted as a function of pressure (mandatory levels only) 
for both RAMS and the Eta model include: a) RMS error and b) bias. The plotting convention is a solid line for 
the RAMS 4-grid errors and a dot-dashed line for the Eta model. 
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1200 UTC Eta/4-grid Cycle 12-h Wind Spd (m/s) at NATL 




Figure 4.33. Vertical profiles of wind speed errors (m s' 1 ) at the national rawinsondes on the Florida 
peninsula for the 12-h forecast, displaying a comparison between the 4-grid configuration of RAMS and the Eta 
model from the 1200 UTC forecast cycle. Parameters plotted as a function of pressure (mandatory levels only) 
for both RAMS and the Eta model include: a) mean observed wind speed, mean RAMS forecast wind speed, 
and mean Eta forecast wind speed, b) RMS error, c) bias, and d) error standard deviation (SD). The plotting 
convention is a solid line for the RAMS 4-grid forecasts, dot-dashed line for the Eta model, and a dashed line 
for observed values. 
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4.2 Subjective Results 

This section summarizes the RAMS subjective verification that was conducted from May through August 
1999. In general, the sea breeze verification results indicate that RAMS has the capability to forecast accurately 
the onset and passage of the ECSB. On the other hand, the subjective precipitation verification indicates that 
RAMS predicted the development and movement of precipitation systems during the 1999 Florida warm season 
with limited skill especially in the 0000 UTC forecasts. 

4.2.1 East Coast Sea Breeze Results 

Table 4.8 contains a contingency table of the forecast and observed occurrences of the ECSB. It 
summarizes the model hits and misses along with the resulting categorical and skill scores. No information on 
the timing of the ECSB is included in Table 4.8. This contingency table strictly focuses on the occurrence of the 
forecast and observed ECSB for a given forecast run on a given day. The maximum possible number of data 
points for Table 4.8 is 170 (2 forecasts per day x 85 possible working days). The 35 missing data points were 
primarily composed of non- work days when the evaluation was not performed and missing/deficient Eta data in 
the initial and boundary conditions. 

The results from Table 4.8 suggest that RAMS does an excellent job in forecasting the occurrence of the 
central Florida ECSB on grid 4. The high probability of detection (0.95) and critical success index (0.92) 
combined with a low false alarm ratio (0.03) support this claim. The HSS provides a benchmark of the model 
performance compared to pure chance (HSS=0). Thus, the HSS of 0.74 suggests that RAMS forecasts of the 
ECSB provide a significant amount of forecast skill. 

It is important to note that these favorable subjective verification results for the occurrence of the ECSB are 
feasible despite a wind direction RMS error as large as 60° (as shown in Section 4.1.1). The subjective 
methodology does not determine the specific direction of the post-sea breeze wind. As long as an onshore wind 
shift occurred or an increase in the onshore wind component was identified at a particular tower, then the 
subjective methodology indicates the presence of a sea breeze passage. Therefore, the wind direction could be 
as much as 90° in error (SE versus a NE wind) but the subjective verification would still indicate a f hit\ 

The methodology used to generate the results in Table 4.8 is somewhat lenient. The verification could be 
made more stringent by classifying a hit as the occurrence of both a forecast and observed sea breeze passage at 
each individual wind tower rather than the occurrence at any tower. To achieve a perfect verification, RAMS 
would have to correctly forecast a sea breeze passage at all towers rather than at any tower according to the 
current methodology. The AMU plans to develop an algorithm to detect sea breeze passages at all KSC/CCAFS 
wind towers for days in which a sea breeze occurred. This modified methodology would increase the sample 
size and the stringency of the subjective evaluation, and allow for the evaluation of the RAMS sea breeze 
predictions at all KSC/CCAFS wind tower sites, not just the 13 towers selected for this study. 

Given all hits among the 13 towers used for verification, the AMU computed the timing errors associated 
with the forecast ECSB. A summary of the forecast ECSB timing errors is given in Table 4.9 for the 0000 UTC 
and 1200 UTC RAMS runs, and for all runs collectively. RAMS output is available only once per hour, thus the 
verification of the ECSB timing at each KSC/CCAFS tower is limited to the nearest hour. Despite this 
limitation, the results shown in Table 4.9 indicate that given the occurrence of both an observed and forecast 
ECSB passage, RAMS predicts the timing quite well. The mean absolute error (MAE), RMS error, and SD are 
all on the order of 1 h, whereas the bias is close to zero for both the 0000 UTC and 1200 UTC forecasts. The 
error statistics at each of the 13 KSC/CCAFS towers do not indicate a correlation between timing errors and 
spatial location with respect to the coastline (not shown). Thus, the timing errors do not suggest a bias over any 
particular portion of the verification domain in Figure 2.1. 
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Table 4.8. A Contingency table of the occurrence of RAMS forecast sea breeze 
versus the observed sea breeze over east-central Florida. Corresponding skill scores are 
listed below. 



Observed Sea Breeze 

No Observed Sea Breeze 

Forecast Sea Breeze 

110 

3 

Sea Breeze Not Forecast 

6 

16 


Probability of Detection: 0.95 False Alarm Rate: 0.03 

Critical Success Index: 0.92 Heidke Skill Score: 0.74 


Table 4.9. A summary of error statistics for the May-August 1999 evaluation 
period are given for the subjective sea breeze timing verification performed on the 13 
KSC/CCAFS tower locations in Figure 2.1. The mean absolute error (MAE), RMS error, 
standard deviation (SD), and bias are shown in units of hours for the 0000 UTC and 1200 
UTC forecast runs, and for all runs collectively. 


0000 UTC 

1200 UTC 

All 

MAE (h) 

0.9 

0.9 

0.9 

RMS error (h) 

1.3 

1.3 

1.3 

SD (h) 

1.3 

1.3 

1.3 

Bias (h) 

-0.2 

0.1 

0.0 


4.2.2 Precipitation Forecast Results 

This section presents the categorical and skill scores of the subjective precipitation verification on grid 4 as 
a function of space and time. Figures 4.34 and 4.35 summarize the categorical and skill scores for the 0000 
UTC and 1200 UTC cycle forecasts respectively, according to the six zones defined in Figure 2.2. Three bars 
are shown in each graph representing the 1-h, 2-h, and 3-h time intervals used for verification of the 
precipitation forecasts. For the 0000 UTC cycle precipitation forecasts, the FAR generally exceeds the POD for 
all verification time intervals in zones 1, 4, 5, and 6 (Figs. 4.34a and b). The highest skill occurs in the inland 
zones (1-3, CSI/HSS between 0.2 and 0.4) whereas the lowest skill is found in the coastal zones (4-6, CSI/HSS 
< 0.2, Figs. 4.34c-d). The greater skill coincident with zones 1-3 is primarily due to a greater occurrence of 
observed precipitation over mainland Florida. Only slight improvement is noted when verifying RAMS 
precipitation forecasts over longer periods of time out to 3 hours. The high FAR combined with the low POD, 
especially over the coastal zones, suggest that the 0000 UTC RAMS forecasts have a tendency to overpredict 
precipitation on grid 4 or to improperly predict the onset and movement of precipitation. It is important to note 
that anomalous forecast precipitation in RAMS may be the cause of a significant portion of the wind and 
temperature errors, especially during the daylight hours. No determination has yet been made regarding the 
nature of the precipitation errors. 

A general improvement over the 0000 UTC forecast cycle occurs in the scores from the 1200 UTC RAMS 
forecasts. For zones 1-3, the POD exceeds the FAR for all verification time intervals, contraiy to the 0000 
UTC precipitation scores (Figs. 4.35a-b). However, the FAR is still generally higher than the POD for the 
coastal zones 4-6. Once again, the skill scores are more favorable for the inland zones (CSI/HSS between 0.3 
and 0.5) rather than the coastal zones (CSI/HSS between 0.2 and 0.4, Figs. 4.35c-d) due to a higher occurrence 
of observed precipitation over the inland zones. 

Despite the higher skill, RAMS has the greatest tendency to overpredict precipitation over the inland zones 
1-3. According to the bias computed from the precipitation contingency tables at each zone (not shown), 
RAMS overforecasts the occurrence of precipitation over the inland zones by 20-49% in both the 0000 and 
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1200 UTC cycles. Meanwhile, the magnitude of the precipitation bias over the coastal zones is smaller than the 
inland zones, despite the lower forecast skill over the coastal regions. The bias for the coastal zones ranges from 
-8 to 17% for both forecast cycles (not shown). 

To support the claim that the verification scores are more favorable in conjunction with peak observed 
precipitation, the categorical and skill scores for both the 0000 UTC and 1200 UTC RAMS forecasts are plotted 
as a function of observation time in Figure 4.36. The verification scores are the highest between 1900 UTC and 
2200 UTC, the typical hours of peak convective activity over east-central Florida during the warm season. 
However, during the morning and evening hours (1300-1700 UTC and 0000-0200 UTC) when observed 
occurrences of precipitation are smallest in number (not shown), the FAR exceeds 0.8 and the POD is generally 
less than 0.3 for the 0000 UTC RAMS forecasts (Figs. 4.36a-c). 

The 0000 UTC FAR once again exceed the POD for nearly all observed times and categories (Fig. 4.36a-c). 
The best skill for all forecasts and valid times is found in the 1200 UTC cycle forecasts valid for the 3-h interval 
from 1900-2100 UTC (Figs. 4.36d-f)* Similar to the verification as a function of zone, little added skill is 
obtained by grouping data into larger time verification bins. Thus, 3-h RAMS accumulated precipitation 
forecasts provide little additional guidance compared to 1-h RAMS precipitation forecasts. Finally, it is 
important to note that anomalous precipitation forecasts may be the cause for a significant portion of the errors 
in winds and temperatures, particularly during the daylight hours. 
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Figure 4.34. Categorical and skill scores for the June- August 1999 subjective precipitation verification of 
0000 UTC RAMS forecasts on grid 4 for each of the six separate zones depicted in Figure 2.3. Each panel 
consists of bar graphs depicting a) the Probability of Detection for 1-h, 2-h, and 3-h verification time intervals, 
b) the False Alarm Rate for 1-h, 2-h, and 3-h verification time intervals, c) the Critical Success Index for 1-h, 2- 
h, and 3-h verification time intervals, and d) the Heidke Skill Score for 1-h, 2-h, and 3-h verification time 
intervals. The left-most (right-most) bars for each zone indicate 1-h (3-h) verification scores according to the 
scales provided. The center bars represent the 2-h verification score. 
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Probability of Detection: 1200 UTC 
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Figure 4.35. Categorical and skill scores for the June-August 1999 subjective precipitation verification of 
1200 UTC RAMS forecasts on grid 4 for each of the six separate zones depicted in Figure 2.3. Each panel 
consists of bar graphs depicting a) the Probability of Detection for 1-h, 2-h, and 3-h verification time intervals, 
b) the False Alarm Rate for 1-h, 2-h, and 3-h verification time intervals, c) the Critical Success Index for 1-h, 2- 
h, and 3-h verification time intervals, and d) the Heidke Skill Score for 1-h, 2-h, and 3-h verification time 
intervals. The left-most (right-most) bars for each zone indicate 1-h (3-h) verification scores according to the 
scales provided. The center bars represent the 2-h verification score. 


61 






Categorical/Skill Scores: 0000 UTC 

1-Hour bins 

□ POD1 ■ FAR1 ■ CS11 ■HSSlI 



13 14 15 16 17 18 19 20 21 22 23 0 1 2 
Hour (UTC) 


Categorical/Skill Scores: 0000 UTC 
2-Hour bins 


□ POP2 1 FAR2 ■ CS12 ■ HSS2 



17 19 21 

Hour (UTC) 


Categorical/Skill Scores: 0000 UTC 
3-Hour bins 

iO POD3 ■ FAR3 ■ CSL3 ■ HSS3] 



16 19 

Hour (UTC) 


Categorical/Sklll Scores: 1200 UTC 
1-Hour bins 

I □ POD1 ■ FAR1 ■ CSU ■ HSS1 



13 14 15 16 17 18 19 20 21 22 23 0 1 2 
Hour (UTC) 


Categorical/Skill Scores: 1200 UTC 

2-Hour bins 

!□ POD2 B FAR2 ■ CS12 ■ HSS2 



17 19 21 

Hour (UTC) 


Categorical/Skill Scores: 1200 UTC 

3-Hour bins 

[□POD3 B FAR3 B CSU B HSS3 1 



16 19 

Hour (UTC) 


Figure 4.36. Categorical and skill scores for 1-h, 2-h, and 3-h bins as a function of valid forecast time for 
both the 0000 UTC and 1200 UTC initialized RAMS forecasts. Each panel consists of bar graphs depicting a) 
the Probability of Detection (POD1), False Alarm Rate (FAR1), Critical Success Index (CSI1), and Heidke Skill 
Score (HSS1) for the 1-h forecast bins from the 0000 UTC RAMS forecasts, b) same as in a), except for 2-h 
forecast bins (POD2, FAR2, CSI2, HSS2), c) same as in a) except for 3-h forecast bins (POD3, FAR3, CSI3, 
HSS3), and d-f) same as a-c), except for the 1200 UTC RAMS initialized forecasts. 
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5. 


Extended Efforts for the 2000 Warm-Season Evaluation 


The ERDAS RAMS evaluation will be extended and enhanced for the upcoming 2000 Florida warm season 
months. First, a subjective procedure will be developed which verifies the first predicted thunderstorm of the 
day. For days with observed thunderstorm activity over the area of grid 4, the AMU will identify the first 
observed thunderstorm via radar reflectivity and the Cloud to Ground Lightning Surveillance System. The first 
predicted ERDAS RAMS thunderstorm will be identified according to the first area of hourly-accumulated 
precipitation on grid 4 in conjunction with mid-level vertical velocities exceeding a particular threshold to be 
determined. 

An additional sea breeze verification will be conducted during the 2000 warm season, extending the efforts 
of the 1999 warm-season evaluation. Once again, GOES-8 visible imagery in combination with radar 
reflectivity will be used to identify the occurrence of an observed sea breeze on any given day. A more 
objective algorithm will be developed which takes into account the subjective decision-making procedure for 
verifying the occurrence of a sea breeze passage at a particular KSC/CCAFS wind tower. The benefit of 
applying a more objective technique is that the sea breeze occurrence and timing can be verified at all 
KSC/CCAFS towers for every successful model run. This methodology will lead to a much larger data set and a 
more stringent verification of the sea breeze passage at each individual KSC/CCAFS tower. Once the validation 
is completed, the sea breeze verification can be calculated for both the 1999 and 2000 warm seasons to include 
all available KSC/CCAFS wind towers. 

The AMU will attempt a stratification of ERDAS RAMS errors for the 2000 warm season according to 
various meteorological regimes and/or model error regimes. For instance, ERDAS RAMS objective errors 
could be compiled for specific low-, mid-, or upper-level wind flows, or various stability regimes. In addition, 
ERDAS RAMS model runs exhibiting similar error characteristics could be grouped together and a particular 
flow or stability regime common to these forecasts (if any) could be identified. 

Finally, convective and grid-scale precipitation forecasts on grids 3 and explicit precipitation forecasts on 
grid 4 will be verified against available rain gauge data in the Florida peninsula. The AMU will obtain archived 
rain gauge data from all available water management districts on the Florida peninsula in order to perform the 
verification. 



6. Summary 


This report contains results from objective and subjective evaluations of the RAMS component of ERDAS 
for the 1999 Florida warm season. The motivation behind this study is that both the 45 WS and 45 SW/SE were 
interested in understanding the representative errors in the upgraded version of RAMS in the ERDAS 
replacement system. Two significant changes occurred in the RAMS portion of the replacement system; a full 
cloud microphysics scheme was implemented on all forecast grids and the innermost nest grid was modified by 
expanding the domain and improving the horizontal resolution from 3 km to 1.25 km. Both the 45 WS and 45 
SW/SE are interested in understanding the RMAS model errors in the upgraded system. Thus, the primary goal 
of this task is to determine the accuracy of RAMS forecasts during all seasons and under various weather 
regimes, concentrating on wind and temperature forecasts that are required for dispersion predictions. 

6.1 Summary of the RAMS 4-grid Configuration 

The characteristics of the RAMS configuration are as follows. 

• RAMS runs in real-time using 4 nested grids with horizontal resolutions of 60, 15, 5, and 
1 .25 km, with the innermost nested grid centered on KSC/CCAFS. 

• The RAMS forecast cycle is initialized every 12 h at 0000 and 1200 UTC. 

• The model generates an initial condition by using the 12-h Eta model forecast as a 
background field and by analyzing local high-resolution data sets including KSC/CCAFS 
wind towers and profilers. 

• RAMS generates a 24-hour forecast that typically takes between 10 and 12 h to complete. 
Occasionally forecasts do not complete the cycle due to extensive convection in the 
model or forecasts cannot be generated because of missing Eta forecasts. 

• RAMS forecast output is available once per hour. 

6.2 Summary of Methodology 

The AMU performed both objective and a subjective evaluations in order to verify RAMS output during the 
1999 warm season months of May- August. The objective component consisted of three segments to compute 
point error statistics. The first segment verified the full 4-grid configuration of RAMS, the second segment 
compared point statistics between the full configuration and a 3-grid configuration of RAMS, and the third 
portion compared RAMS errors against the Eta model errors. 

In the first segment, point error statistics of the full 4-grid configuration of RAMS were calculated. These 
statistics were computed by interpolating forecasts to the locations of a variety of observational sensors. Among 
the sensors used in the objective evaluation were surface METAR observations on grids 1-4, surface buoy data, 
the KSC/CCAFS wind tower network, the KSC/CCAFS 50-MHz DRWP, and all rawinsondes in the Florida 
peninsula including the CCAFS rawinsonde. This segment of the objective component focused on the 
verification of the 0000 UTC RAMS forecast cycle on grid 4 for the following variables and sensors: 

• Temperature and dew point at 1 .8 m for the KSC/CCAFS wind towers. 

• Wind direction and wind speed at 16.5 m for the KSC/CCAFS wind towers. 

• Upper-level temperature, dew point, wind direction, and wind speed at XMR. 

• Upper-level wind direction and wind speed at the KSC/CCAFS 50-MHz DRWP. 

To determine the effects of a decrease in horizontal resolution on the subsequent model errors, the second 
segment of the objective evaluation compared the 4-grid forecast errors to a 3-grid configuration of RAMS. In 
the 3-grid configuration, forecasts were generated by running RAMS using only grids 1-3 and excluding grid 4. 
Model errors were computed for both the 4-grid and 3-grid simulations at the KSC/CCAFS wind towers, surface 
METAR and buoy stations, the KSC/CCAFS 50-MHz DRWP, and the XMR rawinsonde. In order to isolate the 
differences caused by a reduction in horizontal resolution, this report focused on the 4-grid/3-grid comparison at 
the KSC/CCAFS wind towers and XMR, both located within the grid 4 domain. 
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The third segment of the objective evaluation involved a benchmark comparison between the 4-grid 
configuration of RAMS and the Eta model. The AMU archived point forecasts from the 1200 UTC cycle of the 
Eta ffiodel at 14 surface stations across the southeastern United States and Florida, and 4 upper-air stations on 
the Florida peninsula. Eta upper-air forecasts were interpolated to mandatory levels for verification against the 
national rawinsonde data set available from MIDDS. This report focused on the comparison between the 
RAMS and Eta models for the 1200 UTC cycle at TTS, and the four upper-air stations on the Florida peninsula, 
consisting of Jacksonville, Tampa Bay, CCAFS, and Miami. 

The subjective component focused on the verification of the east-central Florida ECSB occurrence and 
timing on grid 4. The AMU verified the forecast ECSB at 13 selected KSC/CCAFS wind towers for both the 
0000 UTC and 1200 UTC cycles during normal working days. A ’hit' was defined as the occurrence of both a 
forecast and observed sea breeze at any of the 13 KSC/CCAFS towers for a given model run. A contingency 
table verifying the occurrence of the forecast ECSB was developed based on the compiled statistics. The AMU 
also verified the tuning of the forecast ECSB to the nearest hour based on the occurrences of both a forecast and 
observed ECSB at a particular KSC/CCAFS tower. 

The AMU also verified the RAMS forecast precipitation on grid 4 by dividing the domain into 6 zones and 
identifying the occurrence of forecast and/or observed precipitation in each zone. The AMU constructed 
additional contingency tables and computed categorical and skill scores based on the compiled statistics. 

6.3 Summary of the ERDAS RAMS 4-grid Objective Evaluation 

This report provided a summary of the surface and upper-air verification at the beginning of Section 4.1.1. 
At the surface, the AMU identified the following: 

• A distinct cool, dry bias occurs in RAMS during the daylight hours within the 
KSC/CCAFS wind-tower network. The maximum biases in the 0000 UTC forecast 
cycle consist of a -3.5 °C temperature and a -2°C dew point bias. 

• The representative wind direction root mean square (RMS) error is 25-50° at 16.5 m, 
composed almost entirely of non-systematic variability and little bias. The largest 
RMS error in wind direction occurs during the nighttime and early morning hours 
associated with light and variable winds. 

• The peak wind speed RMS error is on the order of 2 m s' 1 at 16.5 m, reaching a 
maximum during the day when a +1 ms -1 bias occurs. 

At upper levels, the AMU found the following errors: 

• RAMS consistently generates a forecast temperature profile that is too stable relative 
to the observed XMR soundings. A cool bias of -1 to -2°C occurs in the lower 
troposphere below 650 mb (~ 3.5 km) whereas a warm bias of 0.5°C is found above 
650 mb. 

• The dew point errors are smallest near the surface and at upper levels above 550 mb 
(5 km) and largest at mid-levels (4-5°C in the 600-800-mb layer, or 2-4 km). 

• During the early morning sounding at XMR (1 100 UTC), the largest RMS error in 
wind direction occurs near the surface (65-70°) and decreases with height to 30-40° 
at mid-upper levels. During the late afternoon, post sea breeze sounding, the largest 
RMS error (65-70°) is found at about 850 mb (-T.5 km). The verification of wind 
direction at the 50-MHz DRWP indicates that the largest error occurs in the 2-6-km 
range (30-60°) and decreases with height to about 20-30° above 8 km. 

• A positive wind speed bias up to +2 m s* 1 occurs at upper levels prior to 6 h whereas 
a negative wind speed up to -3 ms* 1 develops after 6 h. At low levels below 900 mb 
( 1 km), a positive wind speed bias up to + 1 .5 m s‘ l occurs. 

A more sophisticated initialization scheme, such as four-dimensional data assimilation, may improve model 
performance and reduce these errors by better incorporating the high-resolution observational data of east- 
central Florida into the RAMS initial condition. 
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6.4 Summary of the ERDAS RAMS 4-grid/3-grid Comparison 

This report summarized the major findings of the 4-grid/3-grid comparison of the 0000 UTC RAMS cycle 
at the beginning of section 4. 1 .2. At the surface, the comparison yielded the following: 

• The most significant difference between the 4-grid and 3-grid errors is that the 3-grid 
RAMS configuration experiences a more pronounced daytime cool, dry bias at the 
surface within the grid 4 domain. RMS error and biases are more than doubled at 
some forecast times in the 3-grid configuration relative to the 4-grid configuration. 

• Only minor differences between the 4-grid and 3-grid configurations occur in the 
forecast wind direction and wind speed. The most substantial difference among these 
variables is that the 3 -grid forecast wind speeds are virtually unbiased for all forecast 
times whereas the 4-grid forecast wind speed exhibit a +1 ms" 1 bias during the 
daytime hours. 

At upper levels, only minor differences were evident with the exception of the low-level temperature and 
dew point forecasts: 

• Consistent with the surface results, the 3-grid forecast temperature and dew point at 
low-levels experience a much larger RMS error and bias. However, at upper levels, 
the 3-grid configuration has a smaller warm bias than the 4-grid forecasts. The 3-grid 
configuration also experiences about a 1-2° moist bias above 600 mb (4 km) 
compared to a virtually unbiased 4-grid dew point forecast at these levels. 

• The wind direction error differences are nominal throughout the upper levels since 
the magnitudes of each model configuration are within 1 0° of each other. 

• Wind speed results are mixed; the 3-grid is less biased below 900 mb (1 km), the 4- 
grid RMS error is smaller at mid-levels (600-800 mb, 2-4 km), and the 3-grid has 
less of a negative bias above 500 mb (5.5 km). 

6.5 Summary of the ERDAS RAMS 4-grid/Eta model Benchmark Comparison 

The beginning of Section 4.1.3 provided a summary of the similarities and differences between the 4-grid 
RAMS configuration and the Eta model for the 1200 UTC forecast cycle at TTS and the Florida rawinsonde 
sites. The AMU found the following results from this comparison at TTS: 

• The Eta model outperforms RAMS in temperature forecasts during the daylight hours 
due to the prevailing cool bias in RAMS. 

• The RAMS and Eta models have a comparable dew point RMS error at TTS. 
However, the Eta model exhibits a +1°C moist bias whereas RAMS is unbiased. 

• Both models have comparable errors in wind direction and wind speed as well. The 
only minor difference noted is that the Eta forecast wind direction experiences a 
positive bias, particularly during the post sea breeze (afternoon) and nocturnal hours. 

In addition, the Eta RMS error in wind speed is about 1 m s" 1 larger than RAMS due 
to a larger positive bias, especially during the nocturnal hours. 

At upper levels, the comparison yielded the following similarities and differences for the rawinsondes in the 
Florida peninsula: 

• Both the RAMS and Eta models generate a temperature profile that is too stable 
relative to the observed sounding. However, the RAMS cool bias at low levels is 
larger than the Eta model. 

• The dew point error profiles are very similar with the exception that the Eta model 
exhibits a low-level warm, moist bias and a mid-level cool, dry bias, whereas RAMS 
is virtually unbiased throughout the troposphere. 
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• The Eta model RMS error in wind direction is about 10-15° less than RAMS mainly 
in the 400-800-mb layer (2-7 km). At low levels, RAMS has a positive wind speed 
bias up to 2 m s' 1 whereas the Eta model is virtually unbiased. All other differences 
between the two models are minor. 

6.6 Summary of the Subjective Evaluation 

Based on the methodology used and the categorical and skill scores, RAMS demonstrated a high level of 
skill in predicting the occurrence of the ECSB with the 13 selected KSC/CCAFS towers. The high probability 
of detection (0.95) combined with the low false alarm ratio (0.03) suggests that RAMS predicts the occurrence 
of the ECSB quite well. In addition, given the occurrence of both a forecast and observed ECSB passage at any 
of the 13 KSC/CCAFS wind towers, RAMS also demonstrated that it can adequately predict the timing of the 
ECSB to the nearest hour. 

The results of the subjective evaluation of forecast precipitation were not as favorable. RAMS 
demonstrated limited skill in predicting the occurrence of warm-season precipitation to the nearest 1, 2, or 3 
hours. In general, the FAR exceeded the POD, especially for the coastal zones and the 0000 UTC forecast 
cycle. The inland verification zones demonstrated greater skill compared to the coastal locations, primarily due 
to a greater occurrence of observed precipitation over the inland zones. The 1200 UTC precipitation forecasts 
were more favorable than the 0000 UTC forecasts, resulting in an improvement in the POD and FAR by about 
0. 1-0.2. By verifying the precipitation forecasts in 2-h or 3-h bins rather than 1-h bins, the categorical and skill 
scores only improved slightly or not at all (generally 0.0-0. 2 improvement), suggesting that ERDAS RAMS 
demonstrates limited skill in predicting grid 4 precipitation to the nearest 3 h. It is important to note that 
anomalous precipitation forecasts may be the cause for a significant portion of the errors in winds and 
temperatures, particularly during the daylight hours. 

The AMU plans to extend and enhance the warm-season evaluation techniques for the upcoming 2000 
warm season. In particular, the AMU will extend the subjective sea breeze verification to include all 
KSC/CCAFS wind towers by developing a more objective technique based on the current subjective 
methodology. In addition, the first thunderstorm will be verified for days in which observed and/or forecast 
convection occurs. Finally, ERDAS RAMS forecasts will be categorized according to specific meteorological 
and/or error regimes in order to diagnose model performance based on these characterizations. 
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Appendix A 

Tables on the Status of RAMS 4-grid, 3-grid, and Eta Forecasts 


Table A1 . The status of RAMS 4-grid, 3-grid, and Eta point forecasts are displayed for May 
1999. An 'X' denotes a completed forecast whereas a blank denotes a missing forecast. 
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Table A2. The status of RAMS 4-grid, 3-grid, and Eta point forecasts are displayed for June 
1999. An 'X' denotes a completed forecast whereas a blank denotes a missing forecast. 
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Table A3. The status of RAMS 4-grid, 3-grid, and Eta point forecasts are displayed for July 
1999. An 'X' denotes a completed forecast whereas a blank denotes a missing forecast. 
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Table A4. The status of RAMS 4-grid, 3-grid, and Eta point forecasts are displayed for August 
1999. An 'X' denotes a completed forecast whereas a blank denotes a missing forecast. 
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Appendix B 

Miscellaneous Verification Figures from the 4-grid Configuration of RAMS 


1200 UTC 4-grid Cycle Temperature (C) for 1.8-m towers 
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Forecast Hour 


Figure Bl. A meteogram plot of temperature errors (°C) for the 1999 warm season months for the 4-grid 
configuration of the 1200 UTC RAMS forecast cycle, verified at the 1.8-m level of the KSC/CCAFS wind-tower 
network. Parameters that are plotted as a function of forecast hour include: a) mean observed temperatures 
(dashed) and mean forecast temperatures (solid), b) RMS error, c) bias, and d) error standard deviation (SD). 
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1200 UTC 4-grid Cycle Temperature (C) for grid 1 Sfc. 
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Figure B2. A meteogram plot of temperature errors (°C) for the 1999 warm season months for the 4-grid 
configuration of the 1200 UTC RAMS forecast cycle, verified at METAR stations located on grid 1. 
Parameters that are plotted as a function of forecast hour include: a) mean observed temperatures (dashed) and 
mean forecast temperatures (solid), b) RMS error, c) bias, and d) error standard deviation (SD). 
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Figure B3. A meteogram plot of dew point errors (°C) for the 1999 warm season months for the 4-grid 
configuration of the 0000 UTC RAMS forecast cycle, verified at METAR stations located on grid 1. 
Parameters that are plotted as a function of forecast hour include: a) mean observed dew point (dashed) and 
mean forecast dew point (solid), b) RMS error, c) bias, and d) error standard deviation (SD). 
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0000 UTC 4-grid Cycle U-Wind (m/s) for 16.5-m towers 


6 



Forecast Hour 


Figure B4. A meteogram plot of u-wind component errors (m s‘ ! ) for the 1999 warm season months for 
the 4-grid configuration of the 0000 UTC RAMS forecast cycle, verified at the 16.5-m level of the KSC/CCAFS 
wind- tower network. Parameters that are plotted as a function of forecast hour include: a) mean observed u- 
wind (dashed) and mean forecast u-wind (solid), b) RMS error, c) bias, and d) error standard deviation (SD). 
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Figure B6. Time-height cross sections of wind direction errors at the KSC/CCAFS 50-MHz DRWP for 
the 1200 UTC RAMS forecast cycle. Parameters that are contoured for every hour include: a) RMS error 
(shaded every 10°) and b) bias (contoured at ± 5°, 10°, 20°, and 30°, with negative values less than -5° shaded). 
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Figure B7. Time-height cross sections of wind speed errors at the KSC/CCAFS 50-MHz DRWP for the 
0000 UTC RAMS forecast cycle. Parameters that are contoured for every hour include: a) mean observed wind 
speed (shaded every 2 m s' 1 ), b) mean forecast wind speed (shaded every 2 m s' 1 ), c) RMS error (shaded every 1 
m s* 1 ), and d) bias (negative values shaded every 1 m s' 1 and positive values contoured every 1 m s' 1 ). 
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Appendix C 

Miscellaneous Verification Figures from the 4-grid/3-grid RAMS Comparison 




Forecast Hour 


Figure Cl. A meteogram plot of the 1200 UTC forecast cycle surface temperature errors (°C) from the 3- 
grid and 4-grid RAMS configurations. Surface temperatures are verified at the 1.8-m level of the KSC/CCAFS 
wind tower network. Parameters that are plotted as a function of forecast hour for both the 4- and 3-grid 
configurations include: a) mean observed temperature, mean 4-grid forecast temperature, and mean 3-grid 
forecast temperature, b) RMS error, c) bias, and d) error standard deviation (SD). The plotting convention is a 
solid line for the 4-grid configuration, dot-dashed line for the 3 -grid configuration, and a dashed line for 
observed values. 
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Figure C2. A meteogram plot of the 1200 UTC forecast cycle surface dew point errors (°C) from the 4- 
and 3-grid ERDAS RAMS configurations. Surface dew points are verified at the 1.8-m level of the 
KSC/CCAFS wind tower network. Parameters that are plotted as a function of forecast hour for both the 4-and 
3-grid RAMS configurations include: a) mean observed dew point, mean 4-grid forecast dew point, and mean 3- 
grid forecast dew point, b) RMS error, c) bias, and d) error standard deviation (SD). The plotting convention is 
a solid line for the 4-grid configuration, dot-dashed line for the 3-grid configuration, and a dashed line for 
observed values. 
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Figure C3. Time-height cross sections of wind direction errors at the KSC/CCAFS 50-MHz DRWP are 
plotted for the 4-grid configuration of the 0000 UTC RAMS forecast cycle. Parameters that are contoured every 
hour include: a) RMS error (shaded every 10°) and b) bias (contoured at ± 5°, 10°, 20°, and 30°, with negative 
values less than -5° shaded). 
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Figure C4. Time-height cross sections of wind direction errors at the KSC/CCAFS 50-MHz DRWP are 
plotted for the 3-grid configuration of the 0000 UTC RAMS forecast cycle. Parameters that are contoured every 
hour include: a) RMS error (shaded every 10°) and b) bias (contoured at ± 5°, 10°, 20°, and 30°, with negative 
values less than -5° shaded). 
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1200 UTC 4-grid Cycle Wind Speed (m/s) 
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Figure C5. Time-height cross sections of wind speed errors at the KSC/CCAFS 50-MHz DRWP are 
plotted for the 4-grid configuration of the 0000 UTC RAMS forecast cycle. Parameters that are contoured every 
hour include: a) mean observed wind speed (shaded every 2 m s' 1 ), b) mean forecast wind speed (shaded every 2 
m s' 1 ), c) RMS error (shaded every 1 m s' 1 ), and d) bias (negative values shaded every 1 m s' 1 ). 
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Figure C6. Time-height cross sections of wind speed errors at the KSC/CCAFS 50-MHz DRWP are 
plotted for the 3-grid configuration of the 0000 UTC RAMS forecast cycle. Parameters that are contoured every 
hour include: a) mean observed wind speed (shaded every 2 m s' 1 ), b) mean forecast wind speed (shaded every 2 
ms ), c) RMS error (shaded every 1 m s* 1 ), and d) bias (negative values shaded every 1 m s’ 1 ). 
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Appendix D 

Miscellaneous Verification Figures from the RAMS/Eta Model Comparison 




Figure Dl. A meteogram plot of the 1200 UTC forecast cycle surface temperature errors (°C) from the 
RAMS 4-grid configuration and the Eta model. Surface temperatures are verified at MCO, the only station on 
grid 3 at which Eta point forecasts are available from MIDDS. Parameters that are plotted as a function of 
forecast hour include: a) mean observed temperature, mean RAMS forecast temperature, and mean Eta forecast 
temperature, b) RMS error, c) bias, and d) error standard deviation (SD). The plotting convention is a solid line 
for the RAMS 4-grid forecasts, dot-dashed line for the Eta model, and a dashed line for observed values. 
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1200 UTC Eta/4-grid Cycle Dew point (C) for MCO 
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Figure D2. A meteogram plot of the 1200 UTC forecast cycle surface dew point errors (°C) from the 
RAMS 4-grid configuration and the Eta model. Surface dew points are verified at MCO, the only station on grid 
3 at which Eta point forecasts are available from MIDDS. Parameters that are plotted as a function of forecast 
hour include: a) mean observed dew point, mean RAMS forecast dew point, and mean Eta forecast dew point, b) 
RMS error, c) bias, and d) error standard deviation (SD). The plotting convention is a solid line for the RAMS 
4-grid forecasts, dot-dashed line for the Eta model, and a dashed line for observed values. 
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1200 UTC Eta/4 -grid Cycle Wind Dir (deg) for MCO 
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Figure D3. A meteogram plot of the 1200 UTC forecast cycle surface wind direction errors (degrees) 
from the RAMS 4-grid configuration and the Eta model. Surface wind direction is verified at MCO, the only 
station on grid 3 at which Eta point forecasts are available from MIDDS. Parameters that are plotted as a 
function of forecast hour include: a) RMS error and b) bias. The plotting convention is a solid line for the 
RAMS 4-grid errors and a dot-dashed line for the Eta model errors. 
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Figure D4. A meteogram plot of the 1200 UTC forecast cycle surface wind speed errors (m s' 1 ) from the 
RAMS 4-grid configuration and the Eta model. Surface wind speed is verified at MCO, the only station on grid 
3 at which Eta point forecasts are available from MIDDS. Parameters that are plotted as a function of forecast 
hour include: a) mean observed wind speed, mean RAMS forecast wind speed, and mean Eta forecast wind 
speed, b) RMS error, c) bias, and d) error standard deviation (SD). The plotting convention is a solid line for 
the RAMS 4 -grid forecasts, dot-dashed line for the Eta model, and a dashed line for observed values. 
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